How Slotstream Runs a 105GB MoE Model on a 16GB Mac at 20 tok/s

Principle: Treating SSD as a Second Memory Tier

The approach relies on a key property of Mixture-of-Experts (MoE) architecture: although the model has 125B total parameters (105GB at 4-bit), only about 6B parameters are activated per generated token. Slotstream exploits this by:

Shared weights stay resident in RAM — the portion used by every layer remains in memory.

Experts are fetched on demand from SSD — when an expert is needed, it is read into a bounded cache.

Hot experts are retained — frequently used experts stay in memory to avoid repeated disk reads.

Expert Lookahead — the engine predicts which experts subsequent layers will need and pre-fetches them from SSD.

The project explicitly states that lookahead only affects speed, not output ; the generated text is identical to full in-memory execution. Cache size influences throughput but never drops experts — the entire model remains available. This effectively replaces the "memory wall" with an "SSD bandwidth problem" via software scheduling.

Benchmark Data: Measured vs. Projected

Official benchmarks from 2025-10-05 on a 48GB M5 Pro with a 33GB memory budget and zero memory swapping across three runs:

Short prompt: 20.91 tok/s

Longer context: 23.69 tok/s

Full coding answer: 21.59 tok/s

Only the 48GB tier is real hardware measurement. Other memory tiers in the table are projected intervals based on the measured curve. Smaller Macs typically have slower SSDs, so actual performance will vary.

An independent data point from Hacker News reports ~12 tok/s hot decode on a 48GB Mac, ~3 seconds cold start to first token, and peak memory of 32GB — notably lower than the official numbers. The author acknowledges that results depend on the specific Mac and SSD.

Prefill (prompt processing) improved significantly in version 0.2.23: 16K prompt latency dropped from 155.22s to 53.94s, and 2K follow-up from 30.73s to 4.42s. Nevertheless, long prompts can still take minutes because data must be streamed from SSD — an inherent cost of the streaming approach.

System Requirements

Chip: Apple Silicon (M-series); Intel Mac, Windows, Linux not supported.

OS: macOS 14 or later.

RAM: 16GB minimum; 8GB is explicitly refused (the tool would rather not run than swap).

Disk: ~ 110GB free space — this is the first hard gate; a 512GB drive is the practical minimum.

Network: Download 88.3GB compressed archive (expands to 105.3GB). Official estimates: 100Mbps ≈ 2 hours, 25Mbps ≈ 8 hours. Real-world tests show bottlenecks at Hugging Face beyond ~400Mbps.

Download is one-time, resumable, and verified. After that, operation is fully offline.

Three Steps to Run

curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh
slotstream doctor
slotstream run --prompt "Why is the sky blue?"

slotstream doctor

reports planned memory usage, estimated speed, and disk sufficiency — letting you decide before downloading the model.

The tool also exposes Ollama, OpenAI, and Anthropic compatible APIs; slotstream launch claude can connect directly to coding agents like Claude Code as a local backend.

Limitations: It Is Not a Universal Solution

Single model only: Currently only Qwen3.8-Flash-Next 4-bit; arbitrary checkpoints cannot be loaded. The project intentionally maintains one model at a time.

Single concurrency: Only one generation task at a time; multiple apps share the same model instance.

Feature gaps: No tool calling, image input, JSON Schema output, logprobs, or embeddings.

Slow prefill: Long prompts (several thousand tokens) require patience as data is pulled from SSD.

Not for 96GB+ Macs: Machines that can fit the model entirely in RAM should use full-resident engines (e.g., oMLX, MLX-Serve) which are faster. Slotstream explicitly does not optimize for large-memory configurations.

Conclusion: The Significance Outweighs the Project Itself

Slotstream validates a trajectory: MoE architecture + high-speed SSD can lower the barrier for frontier-scale open models from "128GB RAM" to "16GB RAM + 512GB disk".

Decision guide:

You have a 16–64GB Mac, want to run a cutting-edge open model locally, and can accept 15–25 tok/s plus tens of GB disk usage — install it and run today.

You have 96GB+ RAM — skip Slotstream; use a full-resident engine.

You need tool calling, multi-model switching, or high concurrency — not suitable yet; wait for future versions.

Final reminder: only the 48GB M5 Pro figure is a real measurement; your SSD speed will cause large variance. Run slotstream doctor before committing to the 110GB download.

References: carloslfu/slotstream (GitHub, MIT); official 2025-10-05 benchmark data; Hacker News community reports.