Running 2.78T-Parameter Kimi K3 on 8GB RAM: Pure C Inference Engine Explained

Project Overview

The GitHub project kimi-k3-in-c by Fareed Khan implements a pure C inference engine for the Kimi K3 model (2.78 trillion parameters) that runs without a GPU or deep learning frameworks. The model weights occupy 1.56 TB on disk, but the engine's peak memory usage is only 8.24 GB, and the compiled binary is just 176 KB.

How It Works

Kimi K3 is a Mixture-of-Experts (MoE) model with 93 layers. Each layer (except the first) contains 896 experts, but only 16 experts are active per token generation step. This means only about 104 billion parameters (3.7% of total) participate in computation per token; the remaining 96.3% stay on disk until needed.

The engine separates the model into two parts:

Experts : Stored in a highly compact format (~0.5 bytes per parameter). The engine computes directly on this compressed representation, avoiding decompression overhead.

Trunk (~113 GB): Contains attention blocks and input/output layers required at every layer. The trunk is repacked into a single 109 GB file with fixed per-layer offsets. At runtime, the engine loads as many trunk layers as fit in memory; the rest are streamed from disk layer by layer.

Combining these techniques reduces the memory requirement from 5.56 TB (naive full-model loading) to 8.24 GB peak.

Memory vs. Speed Trade-offs

The author benchmarked 12 memory configurations from 8 GB to 224 GB. All configurations produce bitwise-identical output. Speed varies:

8 GB: 32.69 seconds per token

224 GB: 19.21 seconds per token

Memory increases 28× while speed improves only 1.7×, indicating the bottleneck is disk I/O. At 8 GB, each token requires reading ~135 GB from disk. A fast NVMe SSD is essential; mechanical drives are impractical.

Counter-intuitively, caching recently used experts does not help because the active expert set changes almost every step. The author found that allocating memory to the always-needed trunk yields a 1.69× speedup over expert caching at the same 128 GB budget.

Verification and Reproducibility

The repository includes a make test target that runs a 13-layer reference model and compares every numeric result against a PyTorch reference implementation, printing ENGINE MATCHES THE REFERENCE EXACTLY on success. This test requires no model download, no network, and no Python — only a few seconds to compile and run.

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j
make test

After verification, the full model (1.56 TB) is downloaded from Hugging Face with a resumable script that validates each shard. The trunk is then repacked into the 109 GB file (~4 minutes). Running the model uses a preset (e.g., laptop) that configures memory allocation:

./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
  --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

On the author's laptop (8.24 GB peak RAM), generating 8 tokens took about 4 minutes and correctly produced "Paris".

Practical Considerations

Disk space: at least 1.7 TB (1.56 TB model + 109 GB trunk), preferably on a local NVMe SSD.

Chinese prompts must be supplied via --prompt-file to avoid shell encoding issues.

Use --incremental for multi-token generation; otherwise the entire prefix is recomputed for each new token.

A helper script measures available RAM and disk speed to recommend the optimal preset.

Who Should Use This

Not suitable for daily chat: ~30 seconds per token, no chat template, no image understanding, base model only.

Valuable for :

Developers who want to study a full LLM inference pipeline without framework abstractions — the entire engine is 7 C files, every step from disk read to matrix multiply is visible.

Researchers or hobbyists with sufficient storage who want a concrete reference for running trillion-parameter models on commodity hardware.

The author notes that the test machine had four GPUs which remained idle; only the CPU and SSD were utilized.

Significance

This project challenges the assumption that models larger than memory cannot be run. By streaming weights from disk and exploiting MoE sparsity, it demonstrates correct full-model inference on consumer hardware, with detailed measurements of memory, data movement, and bottlenecks at every step. It provides a real-world reference for future discussions about local large-model execution.

Source

https://github.com/FareedKhan-dev/kimi-k3-in-c