Running GLM‑5.2 on AMD: 2,626 Tokens/s at Half Nvidia’s Cost

Performance results

Using AMD MI355X, the open‑source GLM‑5.2 model achieved 2,626 tokens per second (node‑level raw throughput) and 213 tokens per second in single‑stream mode, at less than half the cost of an Nvidia B200 solution.

Quantization

Weights were converted from BF16 to MXFP4 with AMD’s Quark tool. Accuracy on three benchmarks changed minimally: GSM8K 96.5 % → 95.5 %, GPQA‑Diamond 92.2 % → 90.3 %, tau2 81.9 % → 83.4 %.

Inference engine selection

vLLM, ATOM and sglang were evaluated. vLLM and ATOM either discarded the quantization benefit or produced degraded long‑text output. sglang preserved output coherence while retaining the speed advantage of the MXFP4 model.

Fixes required for sglang on ROCm

Mixture‑of‑Experts (MTP) layer name mismatch: Quark recorded the shared expert layer with a fixed prefix, while sglang expects a different prefix. Registering the layer under sglang’s name corrected the mismatch and increased single‑stream throughput by ~3×.

Speculative decoding kernel referenced CUDA headers without an ROCm guard, causing compilation failure. Adding a compile‑time guard for ROCm allowed the fused kernel to build.

Configuration tuning

Parallelism changed from TP8 to TP4×DP2.

Low‑precision expert layers were manually mapped to the appropriate AMD kernels, overriding the default slow fallback path.

These adjustments raised node‑level throughput to the reported 2,626 tok/s.

Real‑world workload

For a workload of 20 k input tokens, 1 k output tokens, and a 60 % cache‑hit rate, the system sustained 2.4 requests per second, kept first‑token latency under 2.22 seconds, and achieved 100 % success rate.

Cost comparison

SemiAnalysis measured $0.22 per million tokens for the MI355X at 18 tokens per second per user, versus $0.30 for the Nvidia B200 – a >40 % reduction.

Limitations

SemiAnalysis notes that multi‑node, large‑scale deployments still favor Nvidia’s hardware and software stack; AMD has not yet demonstrated distributed or cross‑node expert parallelism for GLM‑5.2.

References

https://x.com/wafer_ai/status/2073504623374295099?s=20

https://www.wafer.ai/blog/glm52-amd

https://www.wafer.ai/blog/kernels-are-still-the-moat

Code example

[1]https://x.com/wafer_ai/status/2073504623374295099?s=20
[2]https://www.wafer.ai/blog/glm52-amd
[3]https://www.wafer.ai/blog/kernels-are-still-the-moat