Two RTX 3090s Hit 3145 tok/s on 180B Model: Strata Engine Brings Giant LLMs to Gaming PCs

Scorecard: Verified Community Benchmarks

Developer Erman Eroglu (@geldei) reported that Qwen3.8-Flash-Next on 2×RTX 3090 with the Strata engine reaches 3145 token/s prefill at 300K+ context and sustains >120 token/s decode at 400K context. Third-party archive vramcalculator.com logs four runs from the same user showing prefill 2844–3149 tok/s and decode peak 117.3–204.6 tok/s across engine versions 0.1.42 to 0.2.0-beta, with each GPU power-capped at 280W. The dashboard screenshot is AI-generated but the reported metrics (46.7GB VRAM, 475W total GPU power, 64°C, 90% load) match a typical dual-3090 long-context run. Single-3090 community data (9 users) shows decode 52.6–112.2 tok/s and prefill 128–1900 tok/s; doubling via layer splitting yields the observed prefill scaling.

Model: Qwen3.8-Flash-Next — Qwen4 Architecture Preview

Released August 26 by the Qwen team as an experimental preview of Qwen4 (analogous to Qwen3-Next for Qwen3.5). Parameter breakdown:

125B backbone + 51B N-gram embedding table + 4B multi-token prediction head ≈ 180B on disk.

Only 6B parameters activated per token — 512 experts per layer, top-10 routed + 1 shared expert.

48 layers using Gated DeltaNet + Qwen Sparse Attention (QSA) hybrid : linear attention for "recall", QSA micro-block coarse filtering then exact attention on key regions — the architectural root of low long-context latency.

Native 262K context, extendable to 1M; supports text, image, video.

Activating only 6B makes inference fast, but the 180B FP8 weights (~173GB) still need storage — the reason ordinary GPUs cannot hold the model and why Strata's tiered offloading is essential.

Engine: Strata Turns a PC into a Tiered Server

Strata (MIT license, one-click install on Windows/Linux) targets Qwen3.8-Flash-Next on 12–24GB VRAM GPUs + 64GB system RAM . Its tiered strategy:

GPU as hot plate : keeps only the few thousand most-frequently-used "hot experts".

CPU RAM as pantry : holds all 24,576 experts (48 layers × 512), CPU processes the rest in parallel.

SSD as warehouse : stores a large lookup table for on-demand fetch.

Two accelerators boost throughput:

Speculative decoding : a small draft model proposes tokens; the large model verifies in batch — 1.6–1.8× speedup, leveraging the 4B MTP module.

Large-chunk prefill : long inputs ingested in up to 8192-token blocks, combined with QSA sparse attention, pushing prefill well above 1000 tok/s.

Author Niko Veit (@coldniko) is thanked in the tweet; Strata is now the de-facto standard engine for this model. A Chinese tutorial on CSDN (layer split + expert VRAM residency + KV int8) replicates ~3500 tok/s prefill, cross-validating the tweet.

Decision Framework: Mac Studio vs. Gaming PC + Strata

Two cost/performance paths for local LLM inference:

Path A: Mac Studio M5 Ultra — ¥46,999+, 256GB unified memory, 1.2TB/s bandwidth. Runs 4-bit full-weight models with best quality; 70B dense ~15–25 tok/s single-stream. Wins on convenience, silence, one-click ecosystem.

Path B: Gaming PC + Strata — requires 12–24GB VRAM GPU + 64GB RAM; dual 3090 is an advanced config. Trade-offs: quantization loss (community uses IQ3_S ≈ 3-bit), Linux/driver/parameter tuning effort, measurable quality drop vs. full-weight.

Comparison chart (AI-generated) summarizes the contrast. Selection hinges on three factors:

Workload profile : "ultra-long context + frequent repeated queries" (whole-repo code QA, batch document processing). High prefill share amplifies Strata's 3145 tok/s advantage; M5 Ultra's 1.2TB/s bandwidth actually underperforms on prefill throughput.

Quality tolerance : IQ3_S impact on coding tasks is small (community reports), but Chinese writing and complex reasoning suffer perceptibly; M5 Ultra runs 4-bit+ full experts — not comparable.

Willingness to tinker : Strata tuning (power caps, layer splits, speculative params) is technical; Mac's Ollama/LM Studio is plug-and-play.

Bottom line: Want quality + zero hassle → buy Mac; want long-context throughput + recycle old GPUs → run Strata. They solve different battles.

Caveats

Tweet is self-reported; consistent with third-party logs but no full reproducible test report.

300K+ context prefill numbers use IQ3_S quantization, not full quality.

120 tok/s decode at 400K context is "still runs" figure; short-context decode is faster (single-3090 hits 100+ tok/s), but don't benchmark against cloud API concurrency.

Dashboard image is AI-generated — treat as illustrative only.

Closing Thought

The tweet's joke holds the real signal: "What replaces it? Probably Qwen4-Flash-Next." Qwen4's upcoming architecture (6B active + offloadable N-gram table + sparse attention) is designed from the ground up for devices that "can't fit the model." Old GPUs + large RAM + smart engines are pulling the "local LLM" entry barrier from ¥50,000 down to a few thousand yuan. Your next local inference box may not be a Mac.

References: @geldei tweet (2026-11-01); vramcalculator.com Strata benchmark archive; GitHub Strata repository docs; Qwen official Qwen3.8-Flash-Next model card.