Kimi K3.1 Pro Weights Released: 1.56TB MXFP4, 6× Beyond Mac Studio, No GGUF, 0.1 tok/s

Key Takeaways

Cannot run locally, not even close. On November 5, Moonshot AI released Kimi K3.1 Pro full weights as a 1.56TB MXFP4 checkpoint. A 256GB unified-memory Mac Studio falls short by roughly 6× (estimated, excluding KV cache).

No usable download entry point yet. K3.1 Pro replaces K3's full attention layers with Gated DSA, declaring architecture kimi_k31, while llama.cpp mainline only recognizes kimi-k3. As of November 5, Hugging Face returns zero GGUF files, no 1-bit quantization, and Ollama/LM Studio have no loadable files.

Even with enough hardware, speed may deter you. The previous-gen K3 on 4×A100 + 2TB RAM running a 594GB 1-bit GGUF measured 0.10 tok/s — one token every ten seconds, a 500-token reply taking 1 hour 23 minutes. This is K3's measured speed, not K3.1 Pro's, but both share 896 experts and 104B active params, hitting the same memory-bandwidth wall (inferred).

1. Three Numbers: 2.8T, 5.6TB, 1.56TB — Don't Mix Them

Many reports cite "2.8 trillion parameters, BF16 would need 5.6TB." That describes an artifact the vendor never released : K3.1 Pro has no BF16 or FP8 version; MXFP4 (weights 4-bit, activations MXFP8, quantization-aware training) is the native weight. The repo's 96 safetensors shards total ~1561GB, freely downloadable.

Counter-intuitive conclusion: "Quantize further to 4-bit to save half" is a false premise — it's already 4-bit. Going smaller means 2-bit or 1-bit, which is what Unsloth did for K3 with UD-IQ1_S reaching 594GB. That quantization pipeline currently exists only for K3.

Model card specs that matter:

Total params 2.8T , per-token active 104B (~3.7%); 896 routed experts, 16 chosen per token, plus 2 shared experts.

93 layers = 69 KDA + 24 Gated DSA (K3 used Gated MLA), context 1,048,576 , vision encoder MoonViT-V2 401M.

License kimi-k3.1-pro, HF tagged other — not OSI open source, not MIT.

MoE's most common misreading: active params save compute, not memory. The router may pick any expert for the next token, so all 2.8T must reside somewhere. 1.56TB is a "where to place" problem; 104B is a "how much to compute per step" problem — they don't cancel each other.

2. Official Minimum Recipe: 8 Data-Center GPUs Minimum

The model card recommends vLLM, SGLang, TokenSpeed. vLLM's recipe page states baseline plainly: vLLM 0.34.0+, TP8 tensor parallel, two GB300 trays or one B300 node, or one MI355X node , multi-node requires InfiniBand; validation config is GB300 NVL4. TPU path is 4 hosts TP32. The recipe lists MXFP4 variant as 1874GB , noting that on-disk ~1.56TB is only routed-expert weights; attention, shared experts, and vision tower remain BF16 — runtime memory exceeds download size , two different denominators, don't use 1.56TB to calculate VRAM headroom.

Some reports estimate "at least 12 accelerators" using single-card 141GB (H200); official recipe uses 268GB B300, so 8 cards suffice. Both are consistent — just different card capacities. Incidentally, 8×H200 = 1.13TB total, can't even hold weights, let alone KV cache.

Table A: Memory/VRAM summed by per-card rated capacity; items marked "estimated" are not official measurements.

3. Mac Studio 256GB: Capacity Isn't the Only Gap

Table B: tok/s are third-party estimates based on hardware bandwidth, not measurements; vLLM official recipe does not support Mac.

Third-party site vramglass estimated Mac clusters: two 512GB Mac Studios linked via Thunderbolt 5 running MLX or exo, 2-bit ~ 17.7 tok/s ; four nodes 4-bit ~ 10.6 tok/s , total cost ~ $45.8k / $91.6k . All three numbers are estimates , and still assume a K3.1 GGUF exists.

Recalculate your personal budget: 1.56TB ÷ 256GB ≈ 6.1; by vLLM's 1874GB denominator ≈ 7.3; plus system and KV cache → at least 8× 256GB machines (estimated) . But 8 Macs over Thunderbolt doing expert parallelism must shuffle 104B active params' weights per token — bottleneck shifts from capacity to topology , not solvable by linearly throwing money.

What about SSD offloading? Community has experimental engines streaming K3 on 29GB RAM, but at MXFP4 each generated token reads ~ 104B × 0.5 bytes ≈ 52GB (estimated); even at 1-bit ~13GB/token. Mac SSDs are a few GB/s — can't touch 1 tok/s. That 0.10 tok/s above is exactly this wall.

4. 0.10 tok/s: The Number More Deterrent Than "Won't Fit"

This is the most critical section, from a technical log that actually ran K3 (author stresses: speeds below belong only to K3's 1-bit GGUF; K3.1 Pro has no citable measurements ):

Hardware: 4×A100-SXM4-40GB (160GB VRAM) + 2×EPYC 7542 + 2015GiB system RAM .

Model: Unsloth UD-IQ1_S, 594GB / 554GiB, 14 shards, --cpu-moe --no-mmap, load ~9 minutes.

Result: prefill 12.87 tok/s, short prompt 5.5–6.2 tok/s, generation steady at 0.10 tok/s ; llama-bench ran 17 minutes without finishing 128 tokens; GPU utilization during generation 0–1% .

Author explains clearly: bottleneck is memory bandwidth, not compute . Expert weights stay in system RAM executed by CPU; every token must re-fetch the active expert set. Adding GPUs doesn't help unless you can fit the entire model in VRAM. K3.1 Pro keeps the same 896 experts / 104B active, so once GGUF appears and offload path is taken, it will likely hit the same wall (inferred).

So any claim "K3.1 Pro runs at XX tok/s on YY" — ask: measured or bandwidth-calculated? Credible answer today: "not yet measured."

5. Don't Let Benchmarks Drive You: Three Boundaries

Benchmarks are vendor self-tests; strengths and weaknesses sit in the same table. Terminal-Bench 4.0 scores 65.7 (Claude Opus 5.5 66.4, GPT-6 Astra 58.2), SWE Marathon v1.1 51.7, OfficeQA Pro v2 64.4 — all first among six models. But same table shows Terminal-Bench-Science 0.1 only 38.1 (GPT-6 Astra 68.1), HLE-Diamond 39.4, ProgramBench 31.5, hallucination stress-test error rate 8.3% (K3 23.3%, GPT-6 Astra 5.5%). It's an Agent/coding/document model, not universally #1.

⚠️ An unresolved independent audit. Nov 3 R40 Research preprint: in ~ 23% of Terminal-Bench 4.0 attempts, model reasoning referenced instructions not in the task but checked by hidden tests; flipping task rules, 3 of 4 runs still followed hidden rules. Authors suspect training data contamination but explicitly state "cannot prove"; as of writing, no public response from Moonshot. This is exactly the value of open weights: outsiders can probe. Until clarified, that "0.7 points shy of Opus 5.5" isn't a settled verdict.

Agent Arena leaderboard ≠ static benchmarks. "Open-weight #1, overall #13" comes from 17.5k real long-horizon agent sessions, net improvement +5.1% (K3 Max +0.9%), a third-party real-usage metric worth watching, but don't conflate with benchmark scores above.

License and ablation need scrutiny. Kimi K3.1-Pro License grants MIT-style rights: use, modify, fine-tune, distribute, sell all allowed; two red lines only for big players — group MaaS revenue > $20M for 12 consecutive months requires separate agreement; product MAU > 100M or monthly revenue > $20M must display "Kimi K3.1-Pro" prominently; internal use exempt. Additionally, vendor deliberately removed attack/defense training data; internal xdev-bench-hard dropped from original checkpoint's 90.5% to release 81.0%, aiming to keep open version's offense/defense capability at least three months behind frontier closed-source.

Conclusion: Don't Upgrade Your 256GB for This

256GB unified memory's sweet spot is "fits 100GB-class models and runs them," not "big enough." In that tier, Qwen3.8-Flash-Next 4-bit ~111GB (I broke down selection table in previous post) and 70B Q4 ~42GB are the real sweet spots; K3.1 Pro is a full order of magnitude larger.

Follow the money: domestic API pricing ¥13 / ¥1.3 / ¥68 per million tokens input / cache hit / output (international $2 / $10). Four 512GB workstation-class machines estimated $91.6k; at 1 USD ≈ 7.1 CNY, output-only back-of-envelope buys ~ 9.5B output tokens (estimated, excluding power/ops). K3.1 Pro defaults thinking mode on, reasoning intensity max, thinking process counts as output tokens , real bill grows faster than list price; cache hit price is 1/10 of miss, reusing long system prompts is the only clear cost lever.

What's actually worth waiting for: kimi_k31 merged into llama.cpp, GGUF quantization ladder appears; and third parties publish reproducible tok/s on Mac / exo clusters. Until then, this open-weight release's significance is letting cloud vendors and inference providers undercut each other, letting large orgs keep data on-prem — not letting you drag 1.56TB home .

Primary sources: HF official model card | vLLM official deployment recipe | 21st Century Business Herald: compute, license, safety | IT之家: weight release & model card benchmarks | ComputingForGeeks: K3 local measurement & engine status | vramglass: Mac cluster estimates.