ANE on Mac LLMs: 1.14x–1.49x Only for Prefill, M6 Dual 16-Core Changes Game

ANE (Apple Neural Engine) has been present in every Apple Silicon chip since M1, but until recently it was used only for small tasks like image and speech processing. Large model inference ran almost entirely on the GPU. As of November 2024, the Mac local inference community has converged on three distinct ways to put ANE to work, each with different payoffs, hardware requirements, and risks.

Breaking Down "Using ANE" into Three Approaches

Approach 1: Offload prefill only. The engine splits a portion of the feed-forward network (FFN) layers to ANE while the GPU continues to handle decode. Representative projects are Splash 1.3.0 and oMLX. This is the most mature, install-and-use tier.

Approach 2: Put the entire model on ANE. ANEMLL Forge converts models (e.g., Qwen3.8-27B) into Core AI packages ( .aimodel) so that both prefill and decode run on ANE, leaving the GPU largely idle. This is the most aggressive approach and the most sensitive to chip generation.

Approach 3: Keep small models resident on ANE. Background tasks such as speech transcription, embedding retrieval, and rewriting run continuously on ANE, freeing the GPU for the main model. Examples include CoreML-LLM, anemll-embeddings, and WhisperKit. This is the most stable but least flashy tier.

Why Now: Apple Opened Up ANE via Core AI

The bottleneck for five years was not hardware but the software stack. MLX, llama.cpp, Ollama, and LM Studio use Metal/GPU+CPU and had no path to ANE. The turning point came at WWDC 2026: Apple introduced Core AI , replacing the 9-year-old Core ML. Core AI ships with macOS 27 / iOS 27 / watchOS 27 and unifies scheduling across CPU, GPU, and Neural Engine. It natively supports autoregressive generation, streaming output, and multi-turn dialogue. The model format changes from .mlpackage to .aimodel; conversion uses open-source coreai-torch and optimization uses coreai-opt. In short, Core ML was for traditional ML; Core AI is for LLMs and can explicitly dispatch compute to ANE.

The tweet's note that "models must be compiled for specific lengths" reflects Core AI's workflow — Forge's README shows compile --ctx 16K, meaning you compile only the context-length entries you need; changing the length requires recompilation. This is why ANE solutions universally suffer from extremely slow first launch.

Not all ANE projects use Core AI. Splash / oMLX rely on Apple's private ANE interfaces (available on macOS 26.4+). CoreML-LLM uses the legacy Core ML package (iOS 18+ / macOS 15+). Only ANEMLL Forge and anemll-embeddings use the new Core AI (requires macOS 27). The three paths have completely different system requirements.

Approach 1: 1.14×–1.49× Speedup Is Only for First Token

Splash 1.3.0 (Apache-2.0, requires M3 or newer + macOS 26.4+ ; community forks add M1/M2 support) enables GPU/ANE FFN splitting for supported dense models by default. At startup it benchmarks "pure GPU", "share 0.4", and "share 0.8" three times each, takes the median, fits an optimal split point, and only enables ANE if the prediction is at least 5% faster than pure GPU . If the ANE side times out, produces NaN/Inf, or turns out slower, it falls back and re-computes the current chunk on GPU. Disable with splash serve --disable-ane.

The official table below shows cold-start TTFT (time to first token) for Qwen3.8-27B with ~14K token prompts:

Table B: Splash 1.3.0 official TTFT measurements

Two critical caveats when reading this table:

Each row compares only against its own "pure GPU" baseline — hardware, quantization, memory, and power mode differ across rows, so cross-row comparisons are meaningless.

It measures prefill, not decode. The release notes explicitly state: "These are prefill measurements, not decode speedups."

The pattern is clear: the slower the GPU, the larger the relative gain from ANE. M5 Max saves only 1.73 seconds (1.14×), M3 Max in low-power mode saves 28.6 seconds (1.46×), and a community M4 Max test reached 1.77×. If you already have an M5 Max, this path's ceiling is very low.

oMLX provides a counter-example. A developer on M5 Max 128GB measured Qwen3.8-27B oQ4e: assigning 35% of MLP channels and 37.5% of GDN layers to ANE raised 8,193-token prefill from 452 to 510 tok/s (+12.8%). But once Lightning MTP speculative decoding was enabled, the gain vanished — 4K prompt: 505 vs 499 tok/s (nearly flat); 32K context: 422 vs 471 tok/s (actually slower). API-mode first token for 15,100 tokens: 34.8s vs 35.2s. Meanwhile model residency grew from 16.0GB to 20.7GB and load time from 2s to 20s. The author's conclusion: turn it off on this Mac.

Two hidden costs: the split consumes unified memory, reducing automatic context on a 24GB M6 from 73,721 to 61,433 tokens; ANE runs W8A8 heavy quantization, which is not bit-exact with the pure GPU path . First launch also incurs 3–8 seconds of calibration timing plus ~30 seconds of ANE program compilation (cached thereafter at 0.6–1.4s reuse per machine+model+OS version).

The Splash table's M4 Max row uses 14,328 input tokens while others use 14,096, and comes from community testing; M6 acceleration range is 1.46–1.49× (the title uses the upper bound). All are vendor/community self-reported TTFT, not generation speed, not a unified benchmark .

Approach 2: Full Model on ANE — M5 Slower, M6 Is the Watershed

This tier best illustrates that "ANE is not free compute."

Table C: Two full-ANE measurements with completely different methodologies

Upper half (from @luokelabs infographic, M5 Max / Qwen3.8-27B / 2K input): moving the entire model to ANE dropped decode from 39.7 to 22.3 tok/s and prefill from 351 to 147 tok/s , but decode GPU power fell from ~8W to ~ 0.5W . The conclusion is blunt: on M5 this trades speed for power, freeing the GPU for other work, not making inference faster.

Lower half (self-reported by @CowboyCoderHQ, 16GB M6 Mac mini / Ternary Bonsai 2 27B / coding task / 8K context): ANE side 52 vs GPU 36 tok/s , prefill 300 vs 185 tok/s , whole system 12W vs 30W , reproduced on a second M6 mini. This is the first time ANE simultaneously wins on both speed and power.

The hardware explanation aligns: Apple's spec page states M6 has a "dual 16-core Neural Engine" while M5 Pro/Max have a single 16-core . Forge's README adds that M5's ANE compiler rejects FP8 , limiting it to FP16/INT8 kernels, whereas M6 can use FP8 e4m3 8-bit attention — Apple claims this 8-bit attention on M6 reduces whole-system energy per prompt token by ~ 30% compared to the same-chip GPU runtime (single workload, not a general benchmark).

But the barriers are more daunting than the numbers:

192K context release 0.2.1 is M6 only , requiring M6 Mac + 32GB RAM + macOS 27, ~15GB download; full 192K compilation takes ~2h11m, even 16K takes ~8m37s (measured on 32GB M6).

64K release 0.2 supports M5 family , but short-context prefill is only ~15% faster, and M5 first launch must derive its own build (~20s + ~10GB extra disk on M5 Max).

M6 first compile of two function sets takes ~46 minutes (previous version 37 minutes). Switching Python interpreters forces a full recompile because Core AI compilation caches are not portable.

The project states: single workload, three decode repeats, not a general benchmark ; Core AI's cache specialization may silently fall back to GPU — you need inspect_coreai_cache.py --strict to detect it.

The two halves of Table C differ in machine, model, task, context length, and quantization ; they cannot be subtracted horizontally nor used to derive "how much faster M6 is than M5" — only the direction of "ANE relative to local GPU" for each: M5 slower but saves power, M6 faster and saves power. Both are community infographic aggregation and developer self-reports, not official Apple benchmarks.

Approach 3: Small Models Resident on ANE — Most Stable Tier

If you want a "works today, no fuss" answer, this is it: keep speech transcription, embedding retrieval, rewriting, and similar background tasks on ANE permanently, leaving the GPU completely free.

CoreML-LLM (MIT, uses Core ML / .mlmodelc) champions ANE-first, GPU-free. Its published EmbeddingGemma-300M build is only 295MB ; using MLComputePlan it verifies 99.80% operator residency on ANE (1950/1954). Gemma 4 E2B runs at 34.2 tok/s on iPhone 17 Pro (3-chunk path).

anemll-embeddinggemma-2-ane (Core AI) on M4 Pro + macOS 27.0 runs all three towers on ANE: text ~ 35 ms/sentence , whole image ~ 370 ms , audio 50–60 ms.

WhisperKit rewrote speech transcription for ANE, listed as the third resident task in the infographic.

Yet this tier has subtle traps:

ANE compiler does not support fused SDPA. Naive Core ML conversions appear to load without errors but silently fall back to CPU ; only MLComputePlan reveals the true device per operator. Worse, compatibility is still shifting: anemll's EmbeddingGemma 2 explicitly notes that on M5 + macOS 27.2 the ANE pre-check rejects vision and text packages ( Parsing failed, invalid MLIR-MPS program), causing Core AI to fall back to GPU — the same model runs full ANE on M4 Pro but not on M5.

Three Pitfalls Not to Ignore

Compile per fixed length, first launch extremely slow : 8 minutes to 46 minutes depending on context; changing context length requires recompilation; M5 also needs an extra derived build consuming ~10GB additional disk.

Private interfaces + silent fallback : Splash/oMLX depend on Apple's private ANE interfaces — a macOS update can break them; Core ML/Core AI conversion failures throw no errors, they just quietly run on CPU/GPU — you must actively verify residency.

Performance and precision are not free : W8A8 heavy quantization is not bit-exact; splitting eats memory that could hold KV cache; enabling speculative decoding can zero out gains. The official 1.14×–1.49× figures are TTFT, not tok/s.

Which Approach for Your Machine

Table D: Decision by chip tier

One-sentence wrap-up: ANE now truly delivers "free up the GPU" and "save power", not "run LLMs faster"; M6's dual 16-core is the hardware watershed, but its 32GB memory ceiling prevents large models. For most existing Mac owners, the optimal path is to start with Approach 3 to offload small tasks, test Approach 1 on your own machine before enabling — that official 1.14× may be the only gain you see.

Data Sources

Apple official (primary) : M6 "dual 16-core Neural Engine", 12C CPU/12C GPU, 16GB version 153GB/s vs 24/32GB 170GB/s, M5 Pro single 16-core 307GB/s, M5 Max single 16-core 460/614GB/s — support.apple.com/zh-cn/148299, support.apple.com/zh-cn/128108, apple.com.cn/cn/macbook-pro/specs (verified 2026-11-13).

Apple official (WWDC 2026, media & tech docs) : Core AI replaces Core ML, ships with macOS 27 / iOS 27, unified CPU/GPU/ANE scheduling, .aimodel / coreai-torch / coreai-opt / CoreAI.framework, MLX cannot access ANE — developer.apple.com/core-ai, github.com/apple/coreai-models, github.com/apple/coreai-torch, The Paper / 10100 repost of Xinzhiyuan, chenxutan.com technical teardown, HN discussion summary.

Splash 1.3.0 (vendor release notes, self-test) : TTFT table (14,096 tokens, Qwen3.8-27B), M3+ / macOS 26.4+, Apache-2.0, FFN split calibration & fallback, --disable-ane, 24GB M6 context 73,721→61,433, W8A8 non bit-exact, shared prefix 39.6s→6.3s — github.com/incoai/splash release 1.3.0 (newreleases.io mirror), AlphaSignal coverage, GitCode scheduling strategy analysis.

oMLX (third-party measurements) : M5 Max 128GB, 452→510 tok/s (+12.8%), with MTP 4K 505 vs 499 / 32K 422 vs 471, model 16.0→20.7GB, load 2→20s, M3 Ultra dual ANE — eidm.co.za "How to install and tune oMLX on M5 Max 128 GB" (2026-10-13).

ANEMLL Forge (project README, primary) : Core AI package, M5 rejects FP8 / M6 supports FP8 e4m3, M6 8-bit attention ~30% lower energy per prompt token, 192K package M6 only + 32GB + macOS 27, compile 8m37s / 2h11m / 46m, M5 first launch derives ~20s + ~10GB, single workload not general benchmark — huggingface.co/anemll/anemll-forge-qwen3.8-27B and -192K READMEs.

M6 ANE speed & power (developer self-report, non-official) : 16GB M6 Mac mini, Ternary Bonsai 2 27B, 52 vs 36 tok/s, 300 vs 185 tok/s, 12W vs 30W, reproduced on second M6 mini — @CowboyCoderHQ original post, bittide.aicompass.dev repost.

M5 Max full-ANE data (community infographic, non-unified benchmark) : 22.3 vs 39.7 tok/s, 147 vs 351 tok/s, ~0.5W vs ~8W (Qwen3.8-27B, 2K input) — @luokelabs tweet attachment "Mac running LLMs starts using ANE", 2026-11-12 17:45, original image self-notes "numbers from engine authors, community, or independent tests, different methodologies, only rough indication".

Small model resident (project self-reports) : CoreML-LLM EmbeddingGemma-300M 295MB / 99.80% ANE residency, Gemma 4 E2B 34.2 tok/s; anemll-embeddinggemma-2-ane M4 Pro text ~35ms, M5 + macOS 27.2 pre-check rejects and falls back GPU; fused SDPA causes silent CPU fallback, must verify with MLComputePlan — github.com/john-rocky/CoreML-LLM, huggingface.co/anemll/anemll-embeddinggemma-2-ane, huggingface.co/mlboydaisuke/embeddinggemma-300m-coreml.

Topic seed : Luoke Labs @luokelabs tweet thread (2026-11-12 17:45 onward, first tweet 4,266 views), community observation & relay, not an official Apple statement .