Gufo's XDNA2 NPU Offload for LLM Prefill: 5-10% Speedup, Numerical Accuracy Issues

Introduction

The AMD Ryzen AI MAX+ 395 integrates an XDNA2 NPU alongside its Radeon 8060S GPU, yet common local LLM inference frameworks have not utilized the NPU for core model computation. A development branch of Gufo (by mikealanni) implements GPU-NPU cooperative computing, allowing the NPU to participate in Prefill-phase matrix multiplications (GEMM) while the GPU handles the remaining work, aiming to shorten overall Prefill latency through parallel execution.

How NPU Acceleration Works

In the Prefill stage, selected GEMM operations are dispatched to the NPU while the GPU continues other computations; the two execute as concurrently as possible. For Qwen3.8 Flash-Next, the NPU handles portions of the linear attention module's projections, Attention Query and Gate projections, and Shared Expert Gate and Up projections. The model still runs primarily on the GPU; the NPU only takes on tasks suited to its architecture, avoiding a full model migration. The performance gain stems mainly from overlapping GPU and NPU workloads.

Performance Gains

Gufo PR #568 reports initial benchmarks: a 16K token Prefill test shows throughput rising from 1,698 tok/s to 1,821 tok/s (+7.2%). Independent retesting on a Bosgame M5 (Ryzen AI MAX+ 395, 128 GB RAM) with Qwen3.8 Flash-Next UD-Q4_K_XL yields:

16K PP Benchmark: +7.6%

20K real serving request: +4.4%

40K real serving request: +5.5%

Gufo maintainers reproduced ~6-7% PP improvement. A newer experimental branch pushes a 16K test from 1,786 tok/s to 2,125 tok/s (~19%), but this includes additional approximations and optimizations, so the gain cannot be attributed solely to NPU offload nor considered lossless. Overall, GPU-NPU cooperation delivers roughly 5-10% real-world Prefill throughput improvement; Token Generation (TG) speed does not automatically see similar gains because PP is compute-bound while TG is bandwidth-bound.

Accuracy Issues

Enabling the NPU introduces noticeable numerical divergence. Logits distributions differ between pure-GPU and GPU+NPU paths. In PR #568 discussions, fixed-input tests on Qwen3.8 Flash-Next UD-Q4_K_XL produced:

Average KL divergence: 0.0182 (32K context) / 0.0216 (128K context)

Normalized logit error: 0.111 / 0.108

Top-1 token consistency: 93.6% / 93.7%

Top-5 overlap: 89.8% / 90.6%

Top-1 consistency of ~94% means roughly one in 16 predicted positions changes its highest-probability token. In greedy decoding across eight long prompts, seven produced different outputs with NPU enabled. However, a Top-1 mismatch does not directly imply a 6% correctness drop; candidate probabilities are often very close, and the highest-probability token may flip without degrading answer quality. Nevertheless, the NPU path alters the output probability distribution and can affect generated content, so the branch is not recommended for production and remains unmerged.

Error Source Analysis

The discrepancy originates from the NPU kernel's numeric format and accumulation strategy. The original GPU path uses FP16 activations, quantized weights, and FP32 accumulation for certain projections. The experimental NPU branch employs 8-bit Block Floating Point (BFP) and repeatedly rounds intermediate results to BF16 during computation. For a K=2560 matrix multiply, each 64 multiply-accumulate operations trigger a BF16 rounding — 40 roundings total. These errors propagate through subsequent attention calculations and state updates, ultimately shifting the logits distribution. The issue is implementation-specific, not a fundamental NPU limitation; a lossless implementation should be achievable.

Summary and Future Work

The branch demonstrates that the XDNA2 NPU can contribute to LLM Prefill computation on Ryzen AI MAX+ 395, yielding measurable speedups via GPU-NPU parallelism. The primary blocker is reducing compute error while preserving performance. Gufo maintainers propose trying INT8×INT8 with INT32 accumulation on the NPU, starting with Shared Expert layers that already use integer math on the GPU; aligning quantization, accumulation, and scaling schemes could narrow the numeric gap. Additionally, task scheduling, data movement, and synchronization overhead between GPU and NPU offer optimization headroom. Long term, if precise kernels resolve the accuracy problem, the previously idle NPU could become a practical extra compute resource for local LLM inference.

References

Gufo PR #568: XDNA2 NPU Prefill Offload — https://github.com/gufo-org/gufo/pull/568

Gufo Issue #558: NPU Cooperative Computing, Performance & Accuracy Evaluation — https://github.com/gufo-org/gufo/issues/558

mikealanni/gufo experimental NPU branch — https://github.com/mikealanni/gufo/tree/fork-enhancements