vLLM 0.32.0: Segmented Attention & W4A16 Speed Up Ryzen AI MAX+ 395

Segmented Attention: Major Performance Improvement for gfx1151

Ryzen AI MAX+ 395 uses RDNA 3.5 architecture (GPU code gfx1151). While ROCm could run vLLM on these devices, many inference operators were originally developed for Instinct MI series and did not fully exploit consumer RDNA GPUs. Attention was a notable example.

vLLM 0.32.0 adds the ROCM_SEGMENTED_ATTN backend, which reorganizes attention computation on ROCm to reduce compute and memory-access overhead in certain scenarios.

In PR #59132, a user tested Qwen3.8-27B-W4A16-AutoRound on gfx1151. The results show significant throughput gains:

Concurrency 1: 5.49 tok/s → 10.79 tok/s (1.97×)

Concurrency 2: 4.22 tok/s → 8.54 tok/s (2.02×)

Concurrency 4: 3.12 tok/s → 7.76 tok/s (2.49×)

Concurrency 8: 2.77 tok/s → 5.22 tok/s (1.88×)

Enable the new backend with: --attention-backend ROCM_SEGMENTED_ATTN Currently BF16 or FP16 KV Cache should be used for validation. The backend has partial FP8 KV support, but its applicability on gfx1151 remains limited; FP8 KV paths cannot be assumed fully functional.

W4A16 Kernel: Memory-Access Optimization for RDNA 3.5

PR #56301 brings W4A16 improvements. W4A16 stores weights in 4-bit format with 16-bit activations, reducing model memory footprint while retaining numerical precision. However, weight compression does not guarantee efficient matrix computation on GPUs lacking native low-precision matrix instructions. The framework must handle weight unpacking, dequantization, data layout, and matrix multiplication; kernel memory-access patterns directly affect performance.

The optimization adjusts tensor layout and access strides for gfx1151 cache and memory characteristics, reducing patterns that hinder cache utilization.

Benchmark results on Ryzen AI MAX+ 395:

Qwen3-8B W4A16: short-prompt TTFT improved ~15.2%

Llama-3.1-8B W4A16: short-prompt TTFT improved ~22.6%

Tested W4A16 models: decode TPOT improved ~2%–3%

TTFT (Time to First Token) measures prefill latency; TPOT (Time per Output Token) measures decode throughput. The larger prefill gains align with W4A16 inference characteristics: prefill involves more matrix operations where improved data layout enhances compute resource and cache utilization. Decode is more sensitive to weight reads, memory bandwidth, and small-batch efficiency; on unified-memory devices like Ryzen AI MAX+ 395, even improved matrix kernels may not yield proportional decode speedups.

Other ROCm Updates

vLLM 0.32.0 includes a series of ROCm memory-management and kernel optimizations:

PR #59523: ROCm Graph memory offload, remap, and sleep/wake handling. HIP Graph captures computation to reduce repeated kernel-launch overhead, but consumes extra device memory, especially when capturing multiple request shapes.

PR #60021: AMD fused PLE kernels.

PR #59732: QSA fusion.

PLE optimizations reduce intermediate tensors and kernel launches in short convolutions and embedding computations. The author notes these changes primarily target MI300X and MI355X and have little relevance to Ryzen AI MAX+ 395.

FP8 Issue Persists

Despite attention and W4A16 improvements, FP8 remains the biggest problem for vLLM on this hardware. Ryzen AI MAX+ 395 lacks native FP8 matrix compute capability like Instinct MI300/MI350. Loading FP8 weights does not enable efficient FP8 matrix multiplication; the framework must rely on dequantization, type conversion, or custom fused kernels.

The vllm-fp8-strix-halo-kernel-support project attempts to replace unsuitable compute paths with Triton FP8 dequant-GEMV/GEMM. In a specific 24B FP8 model test, output throughput rose from ~1.3 tok/s (BF16 conversion path) to 9.24 tok/s.

Thus, FP8 loading and execution is still the primary limitation when using vLLM 0.32.0 on Ryzen AI MAX+ 395.

Summary

vLLM 0.32.0 makes tangible progress for Ryzen AI MAX+ 395 with optimizations directly targeting gfx1151, providing clear hardware-adaptation value. However, the support arrives late; the author notes that the latest gufo version already achieves 1800+ tokens/s prefill, while vLLM still lags far behind.

Official release: https://github.com/vllm-project/vllm/releases/tag/v0.32.0