How Good Is macOS 27's Built-in `system` Model? A Reality Check

What system Actually Is

system

is the device‑side model entry point exposed by the fm command, not a full model identity. Apple's Foundation Models family includes AFM 3 Core (3B dense) and AFM 3 Core Advanced (20B sparse, 1–4B active), plus cloud models on Private Cloud Compute. The fm → system path is the only confirmed local route; cloud endpoints are separate and not automatically available via the CLI.

Three layers must be distinguished: models Apple owns, models the framework can integrate, and models fm actually exposes. Conflating them leads to overestimation.

Apple's Own Benchmarks — Read by Year

2025 Technical Report: Previous‑Generation Small‑Model Level

Apple's Apple Intelligence Foundation Language Models: Tech Report 2025 publishes MMLU, MMMLU, and MGSM scores for the on‑device AFM (instruct, non‑thinking):

MMLU: AFM 67.85, Qwen‑2.5‑3B 66.37, Qwen‑3‑4B 75.10

MGSM: AFM 74.91, Qwen‑2.5‑3B 64.80, Qwen‑3‑4B 82.97

These show the prior on‑device AFM was competitive among billion‑parameter models but did not universally surpass open peers (e.g., Qwen‑3‑4B leads on these tests). A compression‑optimization comparison in the same report drops MMLU from ~67.8 to 64.4, reminding that paper configurations ≠ deployed versions.

2026 Third‑Generation Evaluation: Generational Improvement Only

Human pairwise preference on general text: AFM 3 Core preferred 45.6% vs. 2025 baseline 23.3% (rest ties). This indicates reviewers more often chose the new model, not a 45.6% accuracy or a 45.6% capability jump. Crucially, it compares two Apple generations, not against GPT/Claude/Gemini flagships. No public cross‑vendor benchmark under identical conditions exists for the current system model.

Third‑Party Data: FoundationModelsBench

An independent project (FoundationModelsBench) recorded on a MacBook Pro (Apple M5, 32 GB, macOS 27.0 beta build 26A5353q, normal thermals, low‑power mode off):

Practical Quick: 25/25 prompts passed (task‑specific pass rate, not general 100% accuracy).

Synthetic Performance: median first‑token latency 0.413 s , median decode throughput 55.35 token/s (excluding tokens already in first cumulative streamed chunk).

One Private Cloud Compute attempt failed before first token.

These numbers reflect one machine, one test set, one measurement method. They demonstrate responsive local interaction and a reproducible methodology, but do not answer gaps in Chinese long‑form, complex code, or multi‑step reasoning versus mainstream models.

Realistic Expectations: A Local Task Component

Treat system as a local task component : well‑scoped inputs, verifiable outputs (e.g., meeting‑note summarization, field extraction from orders, fixed‑label classification). Evidence base: 2025 report = historical small‑model comparison; 2026 intro = generational gain; community sample = concrete speed/task logs. Together they don't compose a full flagship‑level horizontal evaluation.

For cross‑file code edits, hard math, long‑horizon planning, or knowledge‑heavy QA, don't rely solely on "built‑in". Run side‑by‑side tests with your candidate cloud/local models.

Note: fm chat gives a chat UI but adds no web search, repo access, or app‑control permissions. Fluent output ≠ verified output. Chinese capability needs dedicated validation with real Chinese samples; multilingual aggregates don't guarantee stable handling of proper nouns, amounts, dates, or tone.

What fm Can Do Today (macOS 27.0.1, build 26A434 )

Text Generation & Multi‑Turn Chat

fm respond --instructions '用中文回答，保留原文数字。' \
  '把下面的记录整理成三条要点：……'

cat notes.txt | fm respond --instructions '提取待办事项，不补充原文没有的内容。'

respond

for single calls (supports stdin), chat for interactive sessions; both stream.

Structured Output

fm schema object --name Task --string title --boolean urgent > task-schema.json
fm respond --schema task-schema.json \
  '提取任务：明天下午提交报销材料，标记为紧急。'

Schema constrains fields/types; content correctness (dates, amounts) is a separate issue.

Image Input & Built‑in Tools

respond --help

lists --image, --text, and built‑in ocr and barcode tools. Example:

fm respond --image receipt.jpg --tool ocr \
  --text '提取商户、日期和总金额；无法确认的内容明确说明。'

Image understanding, OCR, and barcode are distinct tools; no image generation, speech synthesis, or transcription commands appear in current help.

Local API Server

fm serve --host 127.0.0.1 --port 1976

Exposes /health, /v1/models, /v1/chat/completions (streaming & non‑streaming). Interface similarity ≠ full parameter/endpoint compatibility; verify per integration.

Other utilities: count-tokens, --resume, --save-transcript. Context limits depend on device/runtime; don't assume fixed numbers from older versions.

Can fm Download More Models?

No. Top‑level commands lack pull, download, install, or weight‑import options. respond --help and serve --help show only system under --model. This is direct observation on the current machine.

Distinguish three things:

System‑managed Apple Intelligence asset updates.

Developers integrating other models via Foundation Models framework (requires separate implementations & runtime conditions).

Running open‑weight models yourself with compatible runtimes (memory, quantization, context, runtime support).

None imply fm has a model‑download feature. WWDC26 demos showed fm switching to PCC and using --model for cloud models, but current local help shows only system. Rely on your machine's help for available options; cloud models are not downloaded for offline use.

How to Decide If It's Worth Using

Pick a small real task (50–100 samples with expected outputs). For Chinese extraction, include normal text, missing fields, multiple amounts, ambiguous dates. Require explicit "missing" returns; check for hallucinated completions.

Run system and candidate models on identical inputs, recording:

Task correctness & key‑fact retention.

Format compliance, refusals, timeouts, context overflows.

Request‑to‑usable‑result latency and drift over long runs.

Don't cherry‑pick pretty replies; speed with frequent amount errors vs. slightly slower but stable correctness are different product choices.

For my workflow, fm 's sweet spot is piping local language processing into existing scripts: data stays on‑device, no per‑call cloud API cost, no separate weight deployment. Whether it replaces heavier models for complex tasks must come from same‑condition, real‑task testing.