15 Mac LLM Frameworks Compared: 3-Layer Architecture & Decision Guide for Local Inference

Three-Layer Architecture Overview

The article categorizes 15 mainstream Mac inference frameworks into three layers:

Kernel Engine Layer : The actual compute engines — llama.cpp and Apple MLX (plus MLC LLM for cross-platform).

Desktop Wrapper Layer : GUI/CLI apps that bundle the engines — Ollama , LM Studio , Jan , GPT4All/llamafile .

Service & Cluster Layer : Multi-user, large-model, multi-device serving — SGLang (MLX backend), MLX-Serve , vllm-mlx , exo , plus emerging tools TensorFold , Splash , Slotstream .

Note: Original vLLM requires CUDA and cannot run on Mac Metal; the community port vllm-mlx is the Mac-compatible alternative.

Kernel Engine Layer: llama.cpp vs MLX

llama.cpp — C++ implementation, GGUF single-file format, broad quantization (Q2_K to Q8_0 + IQ series), cross-platform (CPU/CUDA/ROCm/Metal). It is the foundation for Ollama, LM Studio, Jan. On Apple Silicon it uses a Metal translation layer, not native.

MLX / mlx-lm — Apple's native framework, designed for unified memory, compiles ops directly to fused Metal shaders with zero-copy, Python-native, runs Hugging Face Safetensors natively.

MLC LLM — TVM compilation path, targets Web/iOS/Android/desktop from one codebase; relevance on desktop faded after MLX emergence but remains unique for cross-platform deployment.

Real-World Performance Comparison (Community Same-Hardware Benchmarks)

Decode (token generation) : MLX is 15%–45% faster than llama.cpp on dense small models (<14B). On MoE models the gap widens: Qwen3-Coder-30B-A3B on M4 Pro 64GB — MLX ~130 tok/s vs llama.cpp ~43 tok/s (~3×) .

Prefill (prompt processing) : llama.cpp is 1.7–2.2× faster — advantage for long-document / long-context scenarios.

Memory : MLX uses 5%–10% less VRAM .

Summary: MLX generates fast, llama.cpp ingests fast.

Desktop Wrapper Layer: Choose UI, Not Speed

Ollama — CLI + local API ( localhost:11434), MIT license. ollama run pulls models; ideal for Cursor, agents, scripts. Since v0.19, on 32GB+ RAM it auto-enables MLX backend via OLLAMA_USE_MLX=1.

LM Studio — GUI with built-in Hugging Face browser, shows memory-fit warnings. Closed-source, free for personal use. On Mac can select MLX backend; some model sizes run 20%–40% faster than llama.cpp path.

Jan — Open-source GUI (Apache-2.0), actively updated (v0.8.6 as of 2026-10). Choose for open-source compliance.

GPT4All / llamafile — GPT4All targets offline/air-gapped; llamafile is a single binary, zero-install.

Independent tests show speed difference <5% across the three main wrappers running the same model — so pick by interface preference, not raw throughput.

Service & Cluster Layer: For Concurrency, Large Models, Multi-Device

SGLang (MLX backend) — Official Apple Silicon support since 2026. Enable with SGLANG_USE_MLX=1; requires MLX 0.32.0+, PyTorch 2.13.x, macOS 14+. Strengths: structured output, RadixAttention prefix caching, agent serving paradigm. Limitations: no continuous batching, speculative decoding unavailable on MLX backend (official Muse Glimmer test notes DFlash speculative decoding not supported). Official benchmark: M5 Pro, Muse Glimmer 30B q4 — 17.6 tok/s·user, 56.9 output tok/s .

MLX-Serve (oMLX) — Local serving with paged KV + SSD offload, 22k+ GitHub stars, most mature pure-MLX service.

vllm-mlx — Community port bringing vLLM's continuous batching to MLX. arXiv study claims 21%–87% higher throughput than llama.cpp (0.6B–30B) , ~4.3× aggregate throughput at 16 concurrency . Reminder: original vLLM needs CUDA; only vllm-mlx works on Mac.

exo — Clusters multiple Macs / iPhones / workstations into one inference cluster (Apache-2.0). Auto-discovery, Thunderbolt RDMA, tensor parallelism. Official demo: 4× 512GB M3 Ultra running DeepSeek v3.1 8-bit. Scaling data: 2 devices ~1.8×, 4 devices ~3.2× single-device throughput (official benchmarks). For users with multiple devices wanting to run frontier models.

TensorFold — Zig-native MLX inference, focuses on exact speculative decoding, pins MLX 0.31.2.

Splash — Only supports Apple's official packaged format (macOS 26.4+).

Slotstream — Swift-native, SSD-streaming for 125B+ models.

The latter three are 2026 newcomers for early adopters and niche scenarios.

Performance Anchors: Only Verified Numbers

Author's M5 Ultra 256GB tests (previous article): 70B Q4 single-stream 15–25 tok/s ; Qwen3.8 125B MoE 154.5 tok/s ; DeepSeek-V4.1-Flash 552B (3bpw) 92.2 tok/s . These are real measured values, not theoretical peaks.

Community same-machine comparisons : MLX decode +15%–45% (dense small), ~3× on MoE; llama.cpp prefill +1.7–2.2×.

exo official scaling : 2-node 1.8×, 4-node 3.2×.

SGLang official test : M5 Pro, Muse Glimmer 30B q4 — 17.6 / 56.9 tok/s.

Critical : Numbers vary drastically with chip generation, memory size, quantization tier, and concurrency. Any "X is N× faster than Y" claim must be qualified by the exact conditions.

Decision Flowchart: Match Your Workflow

Just chat, no terminal → LM Studio or Jan (Jan for open-source).

Integrate with Cursor / scripts / agents → Ollama (simplest API).

Maximize speed / research / fine-tuning → Directly use mlx-lm (MLX) or llama.cpp.

Multi-user shared API service → MLX-Serve / vllm-mlx / SGLang.

Multiple Macs to build a cluster → exo.

Single machine, need 70B+ models → M5 Ultra + mlx-lm (256GB minimum).

Core principle: Picking the wrong framework costs a few tok/s; picking the wrong workflow wastes real time.

Pitfall Checklist

vLLM doesn't run on Mac — it requires CUDA; use vllm-mlx instead.

Ollama adds wrapper overhead — same machine/model, Ollama can be ~50% slower than raw llama.cpp (tested on M4 Max). For peak speed, run the engine directly.

Quantization formats are not interchangeable — GGUF and MLX are separate ecosystems; switching engines requires conversion (mlx-lm can auto-convert).

Don't trust single benchmarks — first clarify whether your use case is single-user chat or concurrent service, then evaluate speed.

Data sources: Apple official docs, SGLang official docs & tests, exo official benchmarks, r/LocalLLaMA community same-machine comparisons, and author's own M5 Ultra 256GB benchmarks (published earlier). Community/official data not independently reproduced by author; estimates are marked. No fabricated scores.