How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM

GLM‑5.2 is a 744 B‑parameter Mixture‑of‑Experts (MoE) model where each token activates about 40 B parameters, a scale that normally requires high‑end GPUs such as A100/H100.

Colibri shows that the model can run on a pure‑CPU machine with only ~25 GB of RAM by storing expert weights on disk and loading them on demand, while keeping a small dense part resident in memory. The inference engine is written in pure C, consists of a single file ( c/glm.c, ~2400 lines), and has zero external dependencies.

How It Works

The design exploits the MoE architecture. The dense portion (≈17 B parameters) is quantized to int4 and kept in memory, occupying about 9.9 GB. The routing experts—21,504 experts (75 MoE layers × 256 experts + MTP head, each ~19 MB, total ≈370 GB)—are stored on disk. They are streamed in as needed with an LRU cache per layer, and hot experts can be pinned in memory. The operating system’s page cache serves as a free secondary cache.

Implemented Features

Full forward computation matches the reference Transformers implementation under teacher‑forcing.

MLA attention (Multi‑head Latent Attention) compresses each token’s KV cache from 32,768 floats to 576 entries, a 57× reduction. The model uses 64 attention heads without GQA, and MLA alone provides sufficient savings.

Native MTP speculative decoding allows the model to draft multiple tokens in one forward pass. Measured draft acceptance rates are 39‑59% (MTP head uses int8 precision), producing on average 2.2‑2.8 tokens per forward. Rejection sampling guarantees lossless output, so even rejected drafts yield the same result as standard decoding.

Grammar‑constrained speculative drafts enable structured outputs such as JSON or function calls. When a position permits only a single legal byte (e.g., a brace or quote), the grammar itself supplies a draft that is injected as pre‑accepted, without affecting sampling; incorrect grammar merely causes the draft to be rejected.

int8 quantization kernel uses the AVX2 maddubs instruction to implement int8 matrix multiplication, delivering 1.4‑2.5× speed‑up over fp32. Routing chooses the fastest implementation per measurement; single‑row int4 still runs in fp32 when it is faster.

Asynchronous expert prefetching overlaps I/O and computation: while the current batch of experts is being processed, the kernel pre‑reads the next batch of expert weights, maximizing pipeline parallelism.

Getting Started

$ ./coli chat 🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU ✓ pronto in 32s · residente 9.9 GB › ciao!
Ciao! 😊 Come posso aiutarti oggi?

Startup takes about 32 seconds and keeps 9.9 GB resident. The 370 GB of expert files reside on SSD, so SSD speed directly impacts inference latency; NVMe SSDs perform markedly better than SATA SSDs.

Suitable Scenarios

While not a general‑purpose tool for everyday users, Colibri is valuable for:

Local experiments and research : Researchers can run a 744 B‑scale model locally without a GPU.

Offline environments : High‑security or air‑gapped settings can run top‑tier models on a single SSD‑equipped machine.

Exploring niche MoE models : For MoE models lacking existing CPU inference solutions, Colibri’s single‑file C engine offers a reference implementation.

Limitations are clear: CPU inference is slower and unsuitable for real‑time workloads, it requires substantial SSD space (≈370 GB), and frequent expert streaming can accelerate SSD wear.

GitHub: https://github.com/JustVugg/colibri Stars: 2,112+ License: MIT