vLLM Adds FP8 W8A8 Kernel for AMD Strix Halo, Up to 2.54x GEMM Speedup

Why gfx1151 Needs a Dedicated FP8 Kernel

The Ryzen AI MAX+ 395 uses the RDNA 3.5 architecture (gfx1151), which lacks the native FP8 matrix compute capability found in AMD Instinct MI300/MI350 series. Consequently, FP8 weights must be processed via a software path, but the underlying compute efficiency is limited by hardware instructions and kernel implementation. In ROCm/AITER, certain small-batch FP8 GEMM scenarios incur high compute overhead, directly affecting token generation speed during the decode phase of single-user or low-concurrency LLM inference.

PR #63199: HIP Kernel for gfx1151 FP8 W8A8 Blockscale

The pull request implements a HIP kernel targeting gfx1151 that consumes AITER-prepared FP8 weights, activation tensors, and block scales to perform W8A8 computation for specific matrix shapes, outputting BF16 results. The kernel only takes over matrix multiplications matching its supported conditions; all other FP8 ops fall back to the existing AITER path.

Supported Conditions

GPU architecture: gfx1151 (RDNA 3.5)

Quantization format: FP8 W8A8 Blockscale

Batch dimension (M): 1–6

Matrix dimensions: K = 4096, N multiple of 128

Output format: BF16

Unsupported shapes: fall back to original AITER path

The optimization currently targets specific decode projections in models like DeepSeek V4 and does not yet cover general FP8 Linear or MoE operators.

Performance Results: GEMM Speedups up to 2.54×

Developers benchmarked the original AITER path against the new kernel on gfx1151 hardware:

M=1, N=1536: 108.97 μs → 63.98 μs ( 1.70× speedup)

M=1, N=4096: 552.69 μs → 217.71 μs ( 2.54× speedup, 60.6% latency reduction)

M=2, N=4096: 553.67 μs → 266.30 μs ( 2.08× speedup)

These reductions directly lower compute latency for the affected projection layers in small-batch inference.

End-to-End Throughput: ~6.8% Generation Speed Improvement

An end-to-end test ran 10 GSM8K questions generating 1,280 total tokens under Ray TP=2 configuration:

Original: 4.516 tok/s

New kernel: 4.843 tok/s

Improvement: ~6.8%

The end-to-end gain is smaller than individual GEMM speedups because the new kernel only covers a portion of the model's compute; attention, memory access, runtime scheduling, and other matrix ops still consume inference time.

Impact on Ryzen AI MAX+ 395 Users

Community projects (e.g., vllm-fp8-strix-halo-kernel-support) previously attempted Triton-based FP8 dequantization and matmul for gfx1151, requiring custom patches or forks. PR #63199 enters vLLM's upstream review with on-device test data, meaning merged optimizations would flow into official builds and standard deployment pipelines more easily.

Current limitations noted by the author:

Only targets DeepSeek V4 decode projections; no coverage for general FP8 Linear or MoE.

End-to-end test used Ray TP=2; no validation of full DeepSeek V4 FP8 on a single 128 GB Ryzen AI MAX+ 395.

No evidence yet that the PR resolves model loading, GDN correctness, or PLE/Engram memory management for Qwen3.8-Flash-Next FP8.

A related PR #63604 (also draft) addresses a DeepSeek V4 correctness issue when reloading FP8 weights after RL weight sync on MI308X (gfx942), but lacks gfx1151 testing and does not yet prove improved single-395 inference correctness or performance.

Next Steps to Watch

As of 2026-10-26, PR #63199 remains in draft. The author expects future work to extend support to more matrix shapes, Linear and MoE operators, which would make vLLM truly usable on 128 GB Ryzen AI MAX+ 395 systems. Progress is incremental but steady.

References

https://github.com/vllm-project/vllm/pull/63199

https://github.com/vllm-project/vllm/pull/63604