Run 744B MoE Model on 25GB RAM: Colibri's Tiered Storage Breakthrough

Core Architecture: Three-Tier Storage Scheduling for MoE Sparsity

GLM-5.2 is a 744B parameter Mixture-of-Experts (MoE) model from Zhipu AI. Its sparse nature means only ~40B parameters (5.4%) are activated per token, and the actively changing expert weights amount to ~11GB per inference step. Colibri exploits this by splitting weights across three storage tiers:

Dense layers resident in RAM : Attention layers, shared experts, and embeddings quantized to int4 occupy 9.9GB and stay in memory for every forward pass.

Massive experts on NVMe : 19,456 routed experts, int4 quantized, total ~372GB stored on fast NVMe SSD. Only experts selected by the router are loaded on demand.

Pre-fetch + hot cache : The router predicts next-layer experts during current-layer computation, asynchronously pre-fetching them. An automatic heat-map promotes frequently used experts to RAM/VRAM, so speed improves as the conversation progresses.

Multi-SSD mirroring : Two NVMe drives can serve expert shards in parallel (e.g., 9 GB/s + 3 GB/s = 33% bandwidth gain). The mirror drive can hold a subset; unplugging it degrades performance but does not crash.

Five Technical Highlights

1. MLA Attention Compression – 57× KV Cache Reduction

Multi-head Latent Attention compresses KV cache from 32,768 floats per token to 576 floats (57×). The compressed cache can be persisted to disk, enabling exact conversation resume after reboot without re-running prompt prefill.

2. Native MTP Speculative Decoding – Up to 2× Speedup

GLM-5.2 includes Multi-Token Prediction heads that draft 2.2–2.8 tokens per step; the main model verifies them. Critical : the MTP head must be the int8 version ( int8-mtp). Using an int4 MTP head collapses acceptance rate to 0–4%, disabling speculative decoding entirely.

3. Grammar-Constrained Speculative Decoding

When generating structured formats (JSON, etc.), the engine uses grammar rules to predict fixed tokens (braces, quotes, commas) without model inference, boosting throughput for API calls and data extraction.

4. OpenAI-Compatible API Server

Built-in HTTP endpoints support streaming, custom API keys, model IDs, and thinking mode. Three launch modes: ./coli chat – interactive CLI ./coli serve – API only ./coli web – API + visualization dashboard

Windows users run coli.cmd with identical flags.

5. Visualization Panels for MoE Research

Brain : 19,456 experts laid out as a cortical sheet; color = storage tier, brightness = access heat, flashing white = current-step activation.

Atlas : 3D galaxy view clustered by routing affinity; domain-specific experts (poetry, law, Chinese, SQL) form distinct clusters, general experts sit at the center. Fully interactive on a single machine.

Hardware Benchmarks (Same Engine, Same int4 Weights)

6× RTX 5090 (experts fully in VRAM) : 5.8–6.8 tok/s; first token ~13 s, best UX.

128 GB RAM desktop (hot cache) : ~1.8 tok/s; suited for batch offline jobs.

RTX 5070 Ti laptop (GPU+RAM scheduling) : ~1.07 tok/s; balanced.

25 GB RAM laptop (cold start) : 0.05–0.1 tok/s; only proves feasibility; 10–20 s per turn, not daily-driver .

Bottleneck : NVMe random-read performance is the hard ceiling. Minimum 1 GB/s random R/W recommended; VHDX virtual disks significantly degrade speed and are discouraged.

Step-by-Step Deployment Guide

Prerequisites

RAM: 25 GB minimum (usable but slow); 128 GB+ recommended for practical speed

NVMe SSD: ≥372 GB free, high random-read priority

OS: Linux (preferred), macOS, Windows 11 (MinGW-w64, no WSL required), WSL2

1. Get Colibri Engine

Option A – Pre-built binaries (Linux/macOS/Windows):

mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info

Option B – Compile from source (requires gcc/clang + OpenMP):

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh

The core engine is pure C with zero dependencies; Python only serves as launcher and API gateway.

2. Download GLM-5.2 Weights

Fetch the pre-converted int4 shards from Hugging Face, specifically the gs64 + int8-mtp variant (older per-row int4 loses ~9% quality).

Model repo: mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp

To avoid needing 756 GB for the full original checkpoint, use the built-in sharded converter:

./coli convert --model /nvme/glm52_i4

3. Common Launch Commands

COLI_MODEL=/nvme/glm52_i4 ./coli chat      # CLI chat, auto-detects RAM cache
COLI_MODEL=/nvme/glm52_i4 ./coli plan      # Preview weight placement plan
COLI_MODEL=/nvme/glm52_i4 ./coli doctor    # Read-only environment health check
COLI_MODEL=/nvme/glm52_i4 ./coli tune      # Auto-tune and save optimal config
COLI_MODEL=/nvme/glm52_i4 ./coli serve     # Start OpenAI-compatible API
COLI_MODEL=/nvme/glm52_i4 ./coli web       # Dashboard + API, opens browser

Windows example: coli.cmd chat --model D:glm52_i4 Dual-SSD mirroring (optional):

COLI_MODEL=/fast/glm52_i4 COLI_MODEL_MIRROR=/second/glm52_i4 ./coli chat
# COLI_DISK_WEIGHTS=9,3 sets bandwidth ratio

Target Audience & Fit

Good fit for:

Privacy-first workloads (contracts, medical records, internal docs) requiring fully offline inference.

MoE researchers – 2,400 lines of clean C plus Brain/Atlas visualizers enable low-cost routing analysis.

Offline batch processing – 128 GB RAM machines achieve 1.8 tok/s with hot cache.

Open-source enthusiasts exploring frontier local MoE deployment; also supports Kimi K3, DeepSeek V4 Flash, Qwen3.6, etc.

Not suitable for:

Real-time chat on 25 GB RAM – cold-start latency is prohibitive.

Systems without high-speed NVMe – SATA SSDs/HDDs make inference practically unusable.

Closing Perspective

Colibri’s value isn’t the “25 GB runs 744B” headline, but its principled approach: trade speed for capacity without sacrificing precision or altering routing logic, and quantify every quantization loss . A few thousand lines of C unify VRAM, RAM, and disk as a single tiered store, loading experts on demand. It won’t replace vLLM or llama.cpp tomorrow, but it dramatically lowers the entry barrier for flagship MoE models on commodity hardware. The project is MIT-licensed; full documentation and source at https://github.com/JustVugg/colibri.