Splash vs llama.cpp: 5× Speed Claim Tested – Benchmarks, Caveats & Mac Hardware Requirements
The article analyzes Splash, a Mac-specific LLM inference engine claiming 5× speed over llama.cpp, but reveals caveats: the project is not new, "278 models" is a mistranslation of 27B, 210 tok/s only under narrow conditions, model support is limited to a few Qwen variants, and it requires at least 36GB unified memory on M3+ Macs.
Introduction
A screenshot circulating in AI communities claimed that a new open-source inference engine called Splash runs 5× faster than llama.cpp on Mac, achieving 210 tok/s on a 35B MoE model on an M5 Pro. The author fact-checked the claim against Inco AI's official blog (2026-09-17) and GitHub README.
Verification of Five Claims in the Screenshot
"Brand-new engine just born" – False. The project was open-sourced in mid-September 2026, two months before the screenshot circulated.
"278 models" – Mistranslation. The original English text said "a 27B model achieved decoding speeds of 4.5–5.3 times that of llama.cpp"; machine translation misread "27B" as the number 278.
"27B model 4.5–5.3× faster" – True (self-reported). GitHub README shows Qwen3.8-27B (Unsloth UD-Q4_K_M weights) on Metal: M5 Pro 74 vs 16 tok/s (4.6×), M3 Max 92 vs 17 tok/s (5.4×).
"M5 Pro reaches 210 tok/s" – True with missing conditions. Official blog states: 48GB M5 Pro, Qwen3.6-35B-A3B, NVIDIA SPEED-Bench fixed coding task, reasoning enabled, 1024-token output limit, single-request median. Not a general chat speed.
"Fuses 5 speculative decoding methods" – Mismatch. Official sources describe a single DFlash 2 draft model per supported model, compressing propose-verify-accept-state-update into one decode unit. No mention of "5 methods". OpenAI-compatible API and Agent tool support (Claude Code, Codex, OpenCode) are real.
Official Benchmarks
Two benchmark tables are provided:
Table 1 (README GGUF comparison): 27B: M5 Pro 74 vs 16 tok/s, M3 Max 92 vs 17 tok/s (4.5–5.3×). 35B-A3B: M5 Pro 175 vs 69 tok/s, M3 Max 209 vs 66 tok/s (2.5–3.2×). Even with llama.cpp MTP (multi-token prediction), Splash remains 2.7–4.6× faster.
Table 2 (Blog, Agent-style workload vs oMLX on 48GB M5 Pro):
Decoding: 35B short prompt 210 tok/s (1.7×), 27B 74 tok/s (2.0×); at 32K context 143 and 54 tok/s.
Cache replay (32K cached): first token 123 ms (35B) and 282 ms (27B) vs oMLX 816 ms and 2049 ms (6.6–7.3×). Critical for multi-turn Agent conversations.
Concurrency: 4 simultaneous sub-tasks aggregate 357 tok/s (35B) and 170 tok/s (27B), 2–3.9× oMLX. 16 concurrent 32K requests all served vs 9 for generic memory policy.
Note: README reports 175 tok/s for 35B (BF16 KV cache, ANE disabled), blog reports 210 tok/s (standard benchmark). Both are correct under different methodologies; do not mix numerators and denominators across tables.
Why Splash Is Fast: Trading Generality for Speed
Splash inverts the typical runtime approach: instead of one runtime serving thousands of models, it fixes the model first, then builds a dedicated engine around it . Three key techniques:
Shape-specialized Metal kernels generated by an internal kernel agent, pre-compiled for exact model dimensions (4-bit matmul and attention). No on-device compilation or tuning.
DFlash 2 speculative decoding as default path. Small draft proposes token sequences; large model verifies in parallel; accepted state updated directly. Per-request draft state enables coexistence with concurrency and cache reuse. Sliding-window attention keeps long-context memory in check.
Startup memory budgeting. 27B model weights fixed at 15 GiB, draft 1.2 GiB, remainder allocated to KV cache (default 8-bit INT8, optional SSD offload). If it doesn't fit, the engine exits cleanly.
Trade-off: Generality is removed. Unsupported models (Llama, Gemma, DeepSeek) cannot run. Each new model requires retraining a draft model and regenerating kernels.
Four Caveats on the 210 tok/s Claim
Self-tested, no independent reproduction. All numbers from Inco's blog and README. Test prompts are SPEED-Bench coding tasks where DFlash acceptance rates are highest; prose, chat, or long-tail text may show different speedups.
4-bit quantization not free. Engine author Youssofal (MTPLX) notes Qwen3.8's GDN layers are quantization-sensitive; some layers may need 8/16-bit. Splash uses full 4-bit, potential quality cost. KV 8-bit loss deemed small. Inco's benchmark methodology praised as transparent.
"Consistent" ≠ "Identical". README reports next-token agreement: 27B 99.30–99.45%, 35B 97.83–98.14%. This measures next-word prediction similarity to llama.cpp, not task correctness. Differences may accumulate over long generations.
High hardware barrier. Requires M3 or newer, macOS 26.4+, minimum 36GB unified memory for 4-bit models (official recommendation 48GB for headroom). 24GB machines only run smaller GGUF variants. Intel Mac and Windows unsupported.
Should You Switch?
Try Splash if: you have ≥48GB M-series Mac, daily drive Qwen coding Agents, and suffer from memory pressure and long-context cache thrashing. Install: brew install incoai/tap/splash then splash serve (auto-downloads weights and draft). Also available in LM Studio Bionic settings.
Skip if: you have 16/24GB Mac, need other model families (Llama, Gemma, DeepSeek), or only occasionally chat – cloud APIs remain more convenient.
Splash is a radical experiment trading generality for speed on Apple Silicon, proving dedicated engines can extract more performance, but it is not a universal next-gen runtime.
Data sources: Inco AI blog (2026-09-17), GitHub incoai/splash README, LM Studio Bionic integration notes, community teardowns and third-party comments. All tok/s figures are vendor-reported; no independent testing performed. Different tables use distinct hardware, quantization, and sampling methodologies – noted separately and not directly comparable.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
