Hy4 Flash: 224B Params, 12B Active — Can Your Mac Handle It?
The article analyzes Tencent's Hy4 Flash MoE model (224B total, 12B active per token), explaining why activation parameters determine speed but total parameters dictate memory requirements, detailing quantization tiers, official benchmarks on 128GB Mac (44–48 tok/s), 64GB workarounds with Engram offloaded to SSD, and four deployment pitfalls.
Tencent released Hy4 Flash on November 18, providing instruction-tuned, FP8, and base versions. The headline "224B total parameters, only 12B activated per token" led many to ask whether a 32GB Mac could run it. The answer: activation parameters govern speed, but total parameters govern the memory "entry ticket" — all 224B weights must reside in memory for random expert access.
1. Debunking the "12B Activation" Number
MoE works like a clinic with 512 experts; each token invokes only 16 experts (~12B parameters). This activation parameter count determines compute per token and thus generation speed. However, the next token could need any 16 of the 512 experts, so the full 224B weights must be instantly addressable — this is the memory ticket . Hy4 Flash also includes a 63.5B Engram n‑gram embedding lookup table (hashed, outside the main network), which becomes critical for 64GB machines.
Key rule: Activation parameters control speed; total parameters control whether the model fits in memory. Estimating memory from 12B underestimates by nearly 20×.
2. Three Memory Tiers: Check Your Hardware First
Using the rough formula memory ≈ parameters × bytes/parameter, the article compiles a tier list from official announcements, AngelSlim model cards, and community repos (disk sizes include Engram):
Full BF16 — 2 bytes/param, ~440 GB — Enterprise multi‑GPU only
Official FP8 — 1 byte/param, ~287 GB — 8× H200 / 4× B300 minimum
GGUF Q4_K_M — ~4.5 bits/param, ~168 GB total disk (resident ~105 GB + Engram) — 128GB unified‑memory Mac (official supported tier)
GGUF IQ2‑AH — ~2 bits/param, ~125 GB total disk (resident ~72 GB + Engram) — 128GB Mac conservative tier
Community 2.3‑bit layered — mixed precision, resident ~49 GB — 64GB Mac with Engram on SSD
Two reminders: (1) Official GGUF release documents only a 128GB deployment path (128GB Mac or DGX Spark). (2) The Q4_K_M "168 GB" is total disk footprint; actual resident memory is ~105 GB — the difference is the offloadable Engram table, whose handling dictates both speed and quality.
3. Official Benchmarks: 128GB M5 Max Real‑World Speed
AngelSlim (Tencent's compression team) published self‑tested numbers on an M5 Max 128GB using a patched llama.cpp (no third‑party reproduction yet):
Q4_K_M, 4K context: prefill 629.8 tok/s, generation 48.1 tok/s
Q4_K_M, 16K context: prefill 448.1 tok/s, generation 45.2 tok/s
Q4_K_M, 32K context: prefill 418.6 tok/s, generation 44.2 tok/s
Q3_K, 4K context: prefill 743.2 tok/s, generation 50.3 tok/s
Q3_K, 32K context: prefill 520.6 tok/s, generation 47.4 tok/s
Generation throughput 44–50 tok/s — decent for 128GB class, faster than many 70B dense models because only 12B activate per token, but far below smaller models that hit hundreds of tok/s.
Prefill is an order of magnitude faster, yet community reports long‑document ingestion still stalls, especially beyond 32K. Official data stops at 32K; the claimed 1M context has zero local verification .
One community test on RTX 3060 (12GB VRAM) + 32GB RAM + NVMe running an 85GB quant: 20K tokens took ~29 minutes to prefill, then 3.3 tok/s generation — illustrating the gap between "fits" and "usable".
4. 64GB Workaround: Offload Engram to SSD
Community team AtomicChat squeezed Hy4 Flash into ~49GB resident memory for 64GB Macs via layered quantization:
Expert layers: compressed to ~2.3 bits (extreme slimming).
Remaining weights: kept at Q8 to preserve base quality.
Engram table (63.5B): stays on SSD, reading only a few KB per generated token.
Trade‑off: next‑token agreement with full BF16 drops to ~76% (official IQ2‑AH ~78%, Q4_K_M ~89%). Everyday Q&A and image captioning may suffice; complex tasks become unreliable. Hard prerequisites: a custom llama.cpp build from PR #32517 (commit 0a054d0b), because standard llama.cpp, LM Studio, and Ollama cannot load the model; macOS also requires raising the GPU memory limit or it errors out.
5. Four Pitfalls Before You Start
llama.cpp support not yet merged upstream. The official command llama serve -hf AngelSlim/Hy4-flash-GGUF:Q4_K_M fails on stock installs; you must checkout the unmerged PR branch and compile yourself. Toolchain adapters (LM Studio, Ollama) await upstream merge.
GGUF labeled "for end‑to‑end testing only". Tencent's model card explicitly states this — testing ≠ production serving. Official recommendation for commercial deployment is FP8 on 8‑GPU vLLM/SGLang clusters.
License ambiguity. Some sources claim Apache 2.0 for Hy4 series; others say the two formal releases use Tencent Hy License. Check the repository LICENSE file directly before commercial use.
1M context unverified locally. Native 1M token support exists, but all Mac tests stop at 32K. Even with KV cache reduced to 1/6 of Hy4's original, long contexts remain a heavy memory burden.
6. Decision Guide
128GB unified‑memory Mac (M5 Max / M4 Max maxed): Only officially blessed tier. With custom compile, Q4_K_M delivers 44–48 tok/s — suitable for document processing, slides, search. Treat as a productivity tool, not a chatbot replacement.
64GB Mac: Runs only via experimental 2.3‑bit + Engram‑on‑SSD route. Quality loss, custom toolchain, not recommended for primary workloads.
32GB or less: Don't attempt local inference. The gap is not marginal; 128GB is the hard floor.
If you just need the model's capability, not local execution: API pricing is ¥0.5/M input tokens, ¥2/M output tokens, ¥0.02/M cache hits. Free on Tencent WorkBuddy until Nov 30; TokenHub API at 50% off. Calculate monthly API spend vs. a 128GB Mac purchase — that ledger often outweighs spec‑sheet comparisons.
Hy4 Flash's 224B→12B activation is a clever engineering trade‑off, but "how many activate" and "how much memory you need" are separate questions. Don't let the former fool you. Wait for llama.cpp mainline merge and smoother community quantizations before upgrading hardware.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
