Running 744B GLM-5.2 on 25GB RAM: colibri's Disk-Streaming MoE Inference Engine

Memory Hierarchy and Architecture

GLM-5.2 is a 744B-parameter Mixture-of-Experts (MoE) model trained on 28.5 trillion tokens. Only ~5% of parameters (≈40B) are activated per token. colibri splits these activated parameters into two groups: dense layers (attention, shared experts, embeddings) totaling ~17B parameters that stay resident in RAM, and routed experts (~11B per token) that are streamed from NVMe on demand. After int4 quantization, the dense portion occupies 9.9 GB RAM. The 20,000 routed experts (75 MoE layers × 256 experts + MTP head) average 19 MB each, totaling ~370 GB on disk.

The engine treats VRAM (optional hot tier), RAM (warm tier), and NVMe (cold tier) as a unified three-level memory pyramid:

Hot (VRAM, optional): hottest experts, nanosecond latency, sized by budget.

Warm (RAM): dense parameters (9.9 GB) + expert cache, microsecond latency.

Cold (NVMe): all routed experts (~370 GB), millisecond latency.

Prefetching and Caching Mechanisms

To hide disk latency, colibri employs three layered mechanisms:

Router-Lookahead Prefetch: Adjacent MoE layers show 71.6% routing predictability. The engine asynchronously loads the next layer's most probable experts while the current layer computes, overlapping I/O with matrix multiplication.

Learned Cache: Each run records actual expert usage in a .coli_usage file. Frequently used experts are pinned in RAM, so the same engine speeds up with repeated use.

Per-Layer LRU + OS Page Cache: A least-recently-used eviction per layer is backed by the operating system's page cache, providing a free L2 cache layer.

Engineering Details

KV Cache: Uses MLA compression — 576 values per token, ~1/57 of traditional KV cache size, eliminating long-context memory pressure.

Matrix Multiplication: Hand-written AVX2 integer dot-product kernels; no BLAS dependency.

Dual SSD Mirror Mode: On memory-rich machines, two SSDs each hold a full model copy. Read bandwidth doubles; byte-level consistency verified at startup; graceful degradation if one drive fails.

Benchmark Results

Tested on a development machine (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX):

Model load: ~30 seconds; resident RAM: 9.9 GB; peak RSS during chat: ~20 GB.

Cold start: each token reads ~11 GB from disk, yielding 0.05–0.1 token/s.

After cache warm-up and MTP speculative decoding: sustained ≥1 token/s on better hardware; Apple M5 Max reached 1.06 token/s.

MTP speculative decoding produces 2.2–2.8 tokens per forward pass, but the MTP head must use int8 quantization; int4 drops acceptance to 0–4% (the model hub provides an int4-with-int8-mtp variant for this reason).

Community test on RTX 3060: ~0.44 token/s. The tester concluded: "It runs, but usability is another matter."

Deployment and Scaling

Compilation requires only GCC + OpenMP on x86-64 with AVX2; pre-built binaries for Linux, macOS, Windows are provided. The real installation cost is the model: int4 container ~372 GB, requiring ample disk space.

Runtime is a CLI:

$ ./coli chat
  🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
  ✓ pronto in 32s · residente 9.9 GB

Serving interfaces include OpenAI-compatible and Anthropic Messages endpoints; existing clients connect directly. A built-in Brain Dashboard visualizes the 20,000 experts lighting up during inference, aiding routing analysis.

Scaling is linear: more RAM pins more experts via the learned cache; on multi-GPU servers, CUDA_EXPERT_GB=auto PIN_GB=all moves all experts to VRAM, removing disk from the decode path. Since v1.4.0, colibri also supports Kimi K3, OLMoE, and Inkling MoE families.

Comparison with Alternatives

vLLM + GPU cluster: Remains the only choice for high-concurrency, low-latency production traffic. colibri does not target this scenario.

KTransformers: Mature CPU+GPU hybrid for large MoE, but requires Python, PyTorch, CUDA — a much heavier deployment stack.

llama.cpp: Also zero-dependency local inference, but GGUF models must fit entirely in RAM/VRAM. A 744B model cannot fit; colibri trades I/O bandwidth for memory capacity.

colibri's sweet spots:

Air-gapped / classified environments: fully offline, zero dependencies, minimal attack surface (no Python supply chain).

Data-residency private deployments: a single commodity server runs a flagship model.

Low-frequency async workloads: nightly batch processing, offline document analysis where 0.5 token/s is acceptable.

Research & debugging: explore frontier MoE routing behavior locally without renting H100s.

Limitations

Speed: baseline configuration is "writing a letter, not chatting."

SSD wear: high-frequency random reads shorten consumer SSD lifespan — a hidden cost.

Disk footprint: 372 GB model download takes hours.

Single maintainer; performance data from README and community tests only; no third-party audit. Production use requires independent stress testing.

Conclusion

colibri demonstrates that the LLM inference bottleneck can be redefined: instead of stacking VRAM, architectural layering and I/O scheduling bridge a ~30× resource gap, bringing a flagship MoE model into ordinary server rooms. For private deployment selection, the rule remains: avoid for latency-sensitive, high-concurrency services; for offline, classified, low-frequency, or data-local scenarios, it currently has virtually no competition.

Project repository:

https://github.com/JustVugg/colibri