Optimizing Qwen Series Inference with SGLang‑ATOM on AMD Instinct GPUs

Background

Alibaba Mama advertising workloads include agents, image‑text moderation, and offline data generation, primarily using Qwen 3.5/3.6 models such as Qwen3.6‑35B‑A3B and Qwen3.5‑397B‑A17B. Online/near‑line services require strict latency while maximizing throughput; offline jobs prioritize throughput. Input lengths vary from hundreds to hundreds of thousands of tokens, and some requests contain images, creating three core challenges: multi‑model, multi‑business request handling; performance; and engineering delivery efficiency.

Model‑side constraints involve dense, MoE, and multimodal architectures with up to hundreds of billions of parameters, demanding careful weight loading, quantization, attention, MoE, and memory management. Request‑side factors such as long contexts, KV‑cache hit rates, and mixed short‑long request mixes directly affect end‑to‑end latency. Engineering constraints require reusing the existing SGLang serving stack rather than rewriting it for a specific hardware platform.

Solution: SGLang‑ATOM Plugin Mode

ATOM is AMD’s ROCm‑native, lightweight inference engine for Instinct GPUs. It focuses on the model execution layer, exposing AITER kernels, MoE dispatch, MLA attention, and kernel‑fusion optimizations as plugins to open‑source ecosystems such as vLLM and SGLang.

Integration uses SGLang’s out‑of‑tree external model package mechanism:

export SGLANG_EXTERNAL_MODEL_PACKAGE=atom.plugin.sglang.models
python3 -m sglang.launch_server --model-path <model_path> ...

When a request matches a model supported by ATOM, SGLang loads atom.plugin.sglang.models, calls prepare_model(..., engine="sglang"), and builds the ATOM stack via load_model_in_plugin_mode, which handles weight loading, parallel sharding, quantization mapping, and fused checkpoint layout.

The plugin already supports Qwen 3.5, Qwen‑Next, and their FP8 variants. Dense and MoE variants of Qwen 3.5 run as first‑class external models in upstream SGLang, while ATOM provides the execution backend. ATOM also wraps GDN/linear‑attention paths for Qwen‑Next.

1. Multimodal performance optimizations

Host‑side JPEG decoding and tensor conversion cost ~27 ms per 720p image. Offloading JPEG decoding to the GPU via rocJPEG (torchvision backend) reduces latency to ~4 ms—a 7× speedup.

Data transfer between the tokenizer (CPU process) and scheduler (GPU process) previously used gloo:broadcast, which is inefficient for large multimodal blobs. ROCm’s CUDA IPC enables direct GPU‑to‑GPU transfer, eliminating CPU copies and offloading image hashing to the GPU.

ViT encoding is another bottleneck: a 960×1280 image yields ~4,800 visual tokens, leading to >23 million attention operations. Data‑parallel ViT across multiple GPUs yields a 3‑4 % performance gain for a five‑image batch, with larger gains as image count grows.

2. GDN (Gated Delta Network) optimizations

For prefill, ATOM adds a fused‑chunk fast path that merges kernel launches and intermediate tensor reads on AMD GPUs. For decode, a fast path merges gating and recurrent‑state updates, with a fallback to the generic implementation for unsupported shapes. The fast decode path is controlled by an environment variable and includes explicit handling for idle or padding slots.

Accuracy tests on GSM8K show parity with the original implementation, with only minor, acceptable variations.

Additional fused kernels for MRoPE Q/K via Triton further accelerate decode when model shapes permit, while retaining a fallback to the standard rotary embedding.

Benchmark on MI308X shows total token throughput increasing from ~7,467 tok/s to ~8,004 tok/s (+7.2 %), mean end‑to‑end latency decreasing by ~6.5 %, and mean TPOT dropping from ~77.4 ms to ~71.9 ms (‑7.2 %). Prefill fused‑chunk gains add ~3.4 % throughput for Qwen 3.5‑35B‑A3B‑FP8 and ~4.2 % for MoE models.

3. KV‑Cache layout and AITER operators

ATOM integrates AMD’s AITER library, providing high‑performance MHA and PagedAttention operators. The KV‑cache layout is transformed from the standard [num_blocks, num_kv_heads, head_dim, block_size] to a layout aligned with AMD CDNA memory patterns:

k_cache: [num_blocks, num_kv_heads, head_dim // x, block_size, x]
v_cache: [num_blocks, num_kv_heads, block_size // X, head_dim, X]

This eliminates extra device‑to‑device layout conversions during decode, boosting memory‑access efficiency and reducing scheduling overhead. For high‑concurrency, long‑context workloads, decode throughput improves by ~15‑20 % and overall latency drops.

4. Parallel‑strategy selection per business scenario

Online/near‑line services prioritize low tail latency and use a single‑node, high‑TP configuration that splits a model instance across many GPUs to reduce prefill and decode latency. Offline data‑production workloads prioritize throughput and use a single‑node, many‑DP, low‑TP setup that runs many parallel replicas to increase batch size and resource utilization.

Summary

The value of SGLang‑ATOM lies in combining Alibaba’s SGLang‑based scheduling, batching, cache management, multimodal handling, long‑context strategies, and performance analysis (via InferSim) with AMD’s ATOM‑based model adaptation, FP8/PTPC quantization, attention, MoE, kernel fusion, and ROCm communication optimizations. The ROCm‑native path forms a closed optimization loop (ATOM, AITER, MORI, RCCL, Instinct GPUs) that supports current Qwen 3.5/3.6 models and can evolve to Qwen‑Next, multimodal, and longer‑context models.