How a 23k‑Star Project Runs the 744‑Billion‑Parameter GLM‑5.2 on a Single Laptop

colibri is an open‑source, Apache‑2.0 C inference engine designed for MoE large models. It loads only the active experts into memory, keeping the rest on disk, which allows the 744 billion‑parameter GLM‑5.2 model (released by Z.ai) to run on a 12‑core, 25 GB RAM laptop without a GPU.

The MoE architecture activates roughly 400 billion parameters (5.4 % of the model) per token; the dense part (≈170 billion parameters) occupies about 9.9 GB after int4 quantization and stays resident. The 19 456 routing experts, totaling ~370 GB, are stored on SSD and fetched on demand.

To mitigate the latency of disk reads, colibri stores each expert’s three matrices contiguously and reads them with a single pread. The router predicts the next layer’s experts with a 71.6 % hit rate, enabling prefetch while the current layer computes. Frequently used experts are cached in memory and updated each dialogue round.

When a second SSD is available, a mirrored model copy can be placed on it, allowing simultaneous reads that increase bandwidth; the combined 9 GB/s + 3 GB/s yields a 33 % read‑speed improvement over a single disk.

colibri provides three interfaces: ./coli chat – interactive command‑line chat. ./coli serve – launches an OpenAI‑compatible API. ./coli web – a web console showing token speed, latency breakdown, and memory/disk usage, with two visualizations: “Brain” (a live expert‑routing map) and “Atlas” (3D clustering of experts by domain such as poetry, law, SQL).

The KV cache compresses token history 57‑fold (576 floats per token vs. 32 768) and is persisted to disk, allowing conversations to resume after a reboot with identical outputs.

GLM‑5.2 includes an MTP head for speculative decoding; however, the int4 version of the MTP head suffers 0‑4 % acceptance, so only the int8‑MTP version works correctly.

Performance scales with hardware:

6 × RTX 5090 (all experts in GPU memory): 5.8‑6.8 tokens/s, first token ≈13 s.

128 GB RAM CPU desktop: ≈1.8 tokens/s.

Single RTX 5070 Ti: ≈1.07 tokens/s.

25 GB laptop: 0.05‑0.1 tokens/s (demonstrates feasibility, not productivity).

Quick start steps: download the pre‑compiled binary from the GitHub releases page (Linux/macOS/Windows) or compile from source with ./setup.sh (requires gcc/clang with OpenMP). The int4 model file (≈372 GB) is hosted on Hugging Face under mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp. After setting COLI_MODEL to the model path, commands such as ./coli chat, ./coli plan, ./coli doctor, ./coli web, and ./coli serve can be used.

The engine targets privacy‑sensitive scenarios (local processing of contracts, medical records), model researchers (visualizing expert routing), and hobbyists with large‑memory machines willing to tolerate low throughput. The author notes that while the 25 GB demo proves the concept, practical use requires at least 128 GB RAM or GPU acceleration.

Future roadmap includes support for Kimi K2, Qwen3 MoE, and MiniMax; GLM‑5.2 and OLMoE are already functional.

Source code:

https://github.com/JustVugg/colibri