FreeToken Runs a 35B MoE Model on an 8 GB GPU at 39 tokens/s – Is Local Freedom Real?

FreeToken Overview

FreeToken is an open‑source inference engine for Mixture‑of‑Experts (MoE) models that runs large models on GPUs with limited VRAM by keeping all expert weights in system memory and caching only the most recently used experts on the GPU.

Performance Numbers

Qwen3.6‑35B on an 8 GB GPU: 39.3 tokens/s

DeepSeek‑V4‑Flash 284B on a 32 GB GPU: 22 tokens/s

GLM‑5.2 753B on a 96 GB GPU: 14.9 tokens/s

Weight Storage and Routing

A 35 B model requires ~70 GB of 16‑bit weights or ~18 GB when quantized to 4‑bit. FreeToken bypasses the VRAM limit by storing the full weight set in system memory; the GPU holds only a cache of experts selected by the MoE router.

MoE layers contain hundreds of experts, and the router selects a few per token (e.g., Qwen3.6‑35B activates ~3 B parameters per token; DeepSeek‑V4‑Flash selects 6 experts from 256, processing ~13 B parameters). Computation is not the bottleneck; the challenge is locating the selected experts.

Dual Execution Paths

When the router selects an expert absent from the GPU cache, FreeToken can either (1) copy the expert over PCIe for GPU execution or (2) compute it directly on the CPU. Both paths read from the same system‑memory bandwidth pool.

Existing engines fix one path at model‑load time, which fails for token‑level routing changes. FreeToken first measures the bandwidth of both paths on the host machine, then allocates each step’s cache misses proportionally between PCIe transfer and CPU compute. The results from both paths are merged exactly, without approximation.

Hardware‑Specific Scheduling

Profiling reveals divergent optimal schedules: on an RTX 5090 desktop the engine prefers almost all PCIe transfers, while on an 8 GB laptop it offloads most misses to the CPU. Because these preferences cannot be inferred from specifications, FreeToken profiles each machine once before inference.

Memory‑Bandwidth Limitation

GPU compute growth outpaces memory bandwidth. During token generation each weight is read once, yielding an arithmetic intensity near 1 ops/byte. Current GPUs balance around 300 GB/s, making memory bandwidth the dominant factor for generation speed. By treating system memory as a weight warehouse and scheduling on demand, FreeToken sidesteps VRAM capacity limits.

Agent Integration and Incremental Prefill

The engine integrates with an encoding agent that continuously rewrites its history, causing thousands of tokens to be prefixed repeatedly. FreeToken saves checkpoints at agent split points and recomputes only newly added portions. In this scenario the slowest first token takes under 44 seconds , compared with llama.cpp’s peak of 232 seconds and KTransformers’ 946 seconds.

FreeToken also provides OpenAI‑compatible and Anthropic‑compatible APIs, enabling direct calls from Claude, Code, and Codex.

Community Feedback

Some users report that on an RTX 5060 8 GB + 32 GB DDR5 system llama.cpp reaches 34.9 tokens/s, questioning the 39.3 tokens/s claim. Others value the unified CPU‑GPU scheduling and incremental prefilling for agent workloads, while a few express concern about first‑token latency due to CPU offload during prefilling.

Licensing and Resources

FreeToken is released under the Apache 2.0 license. Source code is available at https://github.com/FlashML-org/FreeToken and the accompanying paper is hosted at https://arxiv.org/pdf/2608.16157.