How Strata Runs a 125B Model on a 12GB Gaming GPU

Introduction

Strata is a local inference engine built on llama.cpp/ggml that runs Qwen3.8-Flash-Next (125B MoE) on a standard gaming PC with 12GB VRAM and 32GB system RAM. The repository, created on September 24, 2026, gained 16,595 stars and 1,439 forks in 13 days. The engine is MIT-licensed and installs on Windows via a single START-HERE.bat file.

Three Core Innovations

1. Tiered Scheduling: Turning the Whole PC into VRAM

Qwen3.8-Flash-Next uses a Mixture-of-Experts (MoE) architecture with 24,576 total experts, activating only 10 per token. Strata places the ~4,000 most frequently used experts on the GPU, all 24,576 experts in system RAM, and a 29GB lookup table on the SSD. The official analogy: commonly used spices stay on the countertop; the rest remain in the pantry.

2. Speculative Decoding: Free 1.6-1.8x Speedup

A small helper model predicts the next few tokens; the large model verifies them in parallel. Correct predictions are kept, incorrect ones are recomputed. Final output quality is unchanged because the large model decides each token, but throughput increases 1.6-1.8x. This guess-then-check approach is a mainstream inference acceleration technique.

3. Full API Compatibility for Coding Agents

After starting the local service: /v1 — OpenAI-compatible endpoint (any API key, any model name works) /v1/messages — Anthropic-compatible endpoint (set ANTHROPIC_BASE_URL=http://127.0.0.1:8080 for Claude Code) /v1/responses — Connects to Codex CLI

An MCP server lets AI assistants install and manage Strata itself.

Unified Multi-Tier Memory Architecture

Strata treats the entire computer as a unified multi-tier storage system:

GPU (12-24GB) : Runs model core, holds ~4,000 hottest experts.

RAM + CPU (32-64GB) : Holds all 24,576 experts; CPU computes experts missing from GPU.

SSD : Stores only the 29GB lookup table; each token reads just a few rows.

Benchmarks (Official README)

Tested on RTX 5070 12GB + Ryzen 5 7600 + 64GB RAM (32K prompt / 4K response):

Q2_0 quantization: 94 tok/s generation, 2,650 tok/s prompt processing

IQ3_S quantization: 53 tok/s generation, 1,620 tok/s prompt processing

AMD RX 9070 XT 16GB: Q2_0 achieves 60 tok/s generation. Estimated RTX 3090 24GB: 100-140 tok/s generation.

Key observation: prompt processing (read) is far faster than generation (write) — IQ3_S reads at 1,620 tok/s vs writes at 53 tok/s. This benefits coding agents because feeding code and reading documents are read-heavy, while generated responses are typically short. Long contexts (up to 8,192 tokens) process at 1,000+ tok/s; the first message of ~30K tokens takes ~1 minute, subsequent messages start in seconds.

Installation & Configuration

Hardware check : NVIDIA RTX 20/30/40/50 series or AMD RX 7900/7800/9070/6800/6900 series (≥12GB VRAM), 32GB RAM minimum (64GB for full model), 80GB disk, Windows 10/11 or Linux.

Download repo : Windows double-click START-HERE.bat, Linux run ./setup.sh. Installer auto-detects GPU, asks for model size/context/image support, downloads ~70GB model (supports resume), and starts service.

Open browser at 127.0.0.1:8080 : Built-in Chat, Monitor (GPU load, VRAM, expert cache hit rate), About (config).

Connect agents : Claude Code set ANTHROPIC_BASE_URL=http://127.0.0.1:8080; Cursor/Codex point base URL to 127.0.0.1:8080/v1.

Two gotchas: first launch loads 35-55GB into RAM, freezing the PC for 1-3 minutes — do not close the window. AMD GPUs on Windows do not yet support image input (works on Linux via CPU).

Three Primary Use Cases

Free backend for coding agents : Claude Code / Codex / Cursor API bills are a real cost. Strata's Coder quantization drops half the experts to fit in 32GB RAM; author claims 91% of full-model SWE-bench Verified score. Suitable for daily completions, refactoring, test generation — reserve paid API calls for hard reasoning tasks.

Air-gapped sensitive data processing : Contract review, long-document summarization with image input + long context + data never leaves machine — compliance posture cloud APIs cannot offer. Default single-request concurrency; set parallel: 2 for concurrency (slows each request on 12GB GPU).

Research platform for inference optimization : Tiered scheduling, speculative decoding, CPU/GPU collaboration all open source. Includes a paper ( docs/paper/Strata-Paper.pdf) and community benchmarks ( bench/results, including RTX PRO data from 2026-10-06).

Pros, Cons & Pitfalls

Anonymous author, maintenance uncertain : GitHub account Niko1221 registered 2021, only this repo, bio "AI & fun", no real identity or corporate backing. 13-day-old project with 279 open issues.

No independent reproduction of performance numbers : README speeds are author self-tests. Hacker News commenters reported their own numbers; tech media explicitly stated they hadn't run it themselves. HN report of 100 tok/s on RTX 4090 24GB cannot be extrapolated to 12GB cards.

Model weights license differs from engine : Engine is MIT, but each model has its own license. Qwen3.8-Flash-Next terms must be verified on Hugging Face before commercial or large-scale deployment.

Coder variant weak on Chinese/CJK tasks : Chinese users should choose Q2_0/IQ2_XS/IQ3_S which retain all experts.

Heavy disk and RAM appetite : ~70GB download, 35-55GB RAM at runtime — close browser tabs first. Unsloth's UD-Q4_K_XL on 64GB RAM mostly reads from SSD, dropping generation to 7-8.5 tok/s.

Quantization trade-offs : Q2_0 fast but coarse; IQ3_S finer but slower. Official selection table tiers by RAM — don't pick by feel.

Author's Perspective

Strata's viral growth has genuine technical substance: Qwen team released Qwen3.8-Flash-Next in late August; Strata demonstrably runs it on 12GB VRAM; HN thread (422 comments) includes verified user reports with hardware specs. The "gaming PC runs 125B" gap is reproducible engineering, not marketing.

However, caution is warranted:

Limited algorithmic novelty : Tiered scheduling and speculative decoding are existing llama.cpp ideas; Strata's value is lowering the barrier from compile-and-tune to double-click-a-bat, not a breakthrough.

Trust cost : Anonymous author + 70GB download + full disk access — run in a VM or secondary machine first.

Typical hype cycle : Many such projects follow explode → catch-up → fork-or-fade. No stable version tag yet; premature to replace production API backends.

Recommendation: treat it as a fun local model experiment box to validate your hardware speed. For replacing company API backends, wait for a stable release and third-party benchmarks.

GitHub repository: github.com/Niko1221/Strata