Ascend 910C LLM Inference Optimization: Image & Parameter Tuning for 40% Speedup
Dewu Technology details how they optimized Qwen3.5-35B-A3B inference on Huawei Ascend 910C NPU, cutting TPOT 45% via vLLM-Ascend 0.23.0 upgrade, HCCL offload to AIVector, static kernel precompilation, HBM allocator tuning, jemalloc, operator caching, MoE weight prefetch, HCCL parameters, and cudagraph config, while documenting over ten failed attempts.
Background
With NVIDIA GPU supply constrained and H20 no longer adding new capacity, domestic compute substitution for core business is urgent. Huawei Ascend 910C has been validated for offline batch inference, but online real-time inference demands strict latency (RT) SLAs. Direct deployment of large models on 910C fails to meet SLA, so the team targets lightweight MoE models (Qwen3-32B, Qwen3.5-35B-A3B, Qwen3.6-27B) for smooth online replacement.
Hardware & Model
Hardware: Ascend 910C-560t, single card dual-die (two chips per card, each with 64 GB HBM), 8 cards = 16 dies total.
Primary Model: Qwen3.5-35B-A3B — ~35B total params, ~3B active params, MoE architecture with 30 GDN (Gated DeltaNet) linear attention layers alternating with 10 full-attention layers, shared experts per MoE layer, Gemma-style RMSNorm (weight = 1.0 + weight), and built-in MTP (Multi-Token Prediction) heads for speculative decoding.
Software: vLLM-Ascend 0.22.0rc1 → 0.23.0rc1 → v0.23.0; CANN 9.0.0 → 9.1.0; MindStudio for profiling.
Benchmark Methodology
Metrics: TPOT (Time Per Output Token, decode latency) and TTFT (Time To First Token). Baseline comparison: H20 single card (vLLM 0.23.0).
Load: Input 1024 tokens, output 512 tokens (min/max locked), concurrency c=1/4/8.
Datasets: Random dataset for baseline (eliminate cache/distribution effects), later real business dataset and GSM8K.
Profiling: Ascend official profiler + MindStudio to capture NPU/host timelines, operator latency, communication, host gaps per decode step.
evalscope perf \
--parallel 1 4 8 \
--number 40 80 80 \
--warmup-num 20 \
--model <model_name> \
--url http://127.0.0.1:5001/v1/chat/completions \
--api openai \
--stream \
--dataset random \
--max-tokens 512 --min-tokens 512 \
--min-prompt-length 1024 --max-prompt-length 1024 \
--prefix-length 0 \
--tokenizer-path <path/to/model>/qwen3.5-35b-a3b \
--outputs-dir "./my_output"Baseline & Bottleneck Analysis
On 910C-752t (TP=2), vLLM-Ascend 0.22.0rc1, cudagraph enabled, random data, input 1024/output 512:
TTFT: Ascend outperforms H20 (~24x faster prefill).
TPOT: H20 outperforms Ascend (H20 ≈ 0.60–0.65× Ascend). Decode is the primary optimization target.
MindStudio profile of c=1 decode step (40 decoder layers, each with GDN/Full-Attn + MoE, thousands of kernel launches) reveals six major overhead categories:
Host scheduling & dispatch: Microsecond kernels; host-side kernel dispatch, aclnn execution, event record/wait dominate. Example: triton slot_mapping kernel ~131 µs launch overhead; per-layer conv weight transpose+contiguous ~4.3 µs × 30 layers ≈ 129 µs/step.
Host-device sync & redundant copies: Speculative decoding verification path uses numpy on host to rebuild masks, copies from paged memory (non-blocking copy actually synchronous), breaking async pipeline (~30% of step time). model_runner also has unconditional copy_to_gpu/fill_ each step.
Collective communication: TP=2 requires multiple AllReduce per layer. Default HCCL runs on AICPU (weaker than AI Core, hard to overlap with compute). compute_logits does all-gather of full vocab logits (O(batch × vocab_size)) while greedy sampling only needs argmax.
HBM bandwidth (MoE weight loading): At c=1, GroupedMatmul is bandwidth-bound; expert weights loaded from HBM form critical path; shared expert weight loading serial with routed expert compute.
Graph compilation quality: Dynamic-shape aclnn kernels still invoke runtime GetWorkspaceSize at replay; static kernel precompilation can bake this in.
Memory allocation: Host glibc malloc shows lock contention under high-frequency small allocations; device HBM allocator page granularity affects fragmentation and mapping overhead.
Optimization strategy: easy wins first — version upgrades, env vars, then source-code changes, finally operator fusion.
Optimization Roadmap (Part 1: Image & Parameter Tuning)
All items here require no source-code changes; highest ROI.
1. Inference Framework Upgrade
Switching from vLLM-Ascend 0.22.0rc1 to 0.23.0rc1 (all else equal) dropped c=1 TPOT from 14.2 ms to 12.0 ms ( -15.4% ). Gains from community improvements in MoE communication, graph compilation, scheduler. Nightly builds added only ~0.1 ms. Lesson: upgrade early, once, to latest stable ; all subsequent source-level work based on v0.23.0.
CANN 9.0.0 → 9.1.0 further reduced c=1 TPOT from 8.8 ms to 8.5 ms ( -3.4% ) and is prerequisite for later SuperKernel stream fusion.
2. Collective Communication Offload to AIVector: HCCL_OP_EXPANSION_MODE=AIV
Default HCCL collective ops (AllReduce) run on AICPU, serializing with compute. Setting export HCCL_OP_EXPANSION_MODE=AIV moves them to AI Vector units.
Qwen3.5-35B-A3B (0.23.0rc1 baseline): c=1 TPOT 12.0 ms → 11.2 ms ( -6.6% ).
Qwen3-8B: 9.9 ms → 9.25 ms ( -6.5% ), stable gain.
Community later enabled same for specific models (e.g., MiniMax-M2), confirming generality.
3. Static Kernel Precompilation
vLLM-Ascend graph mode uses dynamic-shape aclnn kernels; even with cudagraph capture, replay still runs GetWorkspaceSize etc. Static precompilation bakes workspace sizes for each cudagraph_capture_size at startup.
Enable (multi-card requires LOCAL_WORLD_SIZE):
export ASCEND_RT_VISIBLE_DEVICES=0,1
export LOCAL_WORLD_SIZE=2
vllm serve /root/workspace/models/Qwen3-32B \
--tensor-parallel-size 2 \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--additional-config '{"ascend_compilation_config": {"enable_static_kernel": true}}'Gain model-size dependent: Qwen3-32B: 31.5 ms → 30.9 ms ( -1.9% ); Qwen3.5-35B-A3B (after HCCL AIV): 11.2 ms → 9.1 ms ( -18.7% ) because smaller active params → more numerous tiny kernels → higher host dynamic-dispatch overhead.
Pitfall: compilation invokes CANN toolchain; missing cann_compat.h in container fixed by inlining compat header in launch script (detailed in Part 2).
4. HBM Allocator Tuning: Virtual Memory + pin_memory + page_size
torch_npuallocator controlled via PYTORCH_NPU_ALLOC_CONF. For online low-concurrency:
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True,base_addr_aligned_kb:16,page_size:1g,pinned_use_background_threads:True,pin_memory_expandable_segments:Trueexpandable_segments:True: virtual memory mapping, segments grow on demand, reduces fragmentation, large tensors no longer need physical contiguity. page_size:1g: 1 GB pages reduce page-table entries and mapping overhead. pinned_use_background_threads:True / pin_memory_expandable_segments:True: pin-memory allocation offloaded to background threads, avoids blocking inference main thread; pin pool expandable. base_addr_aligned_kb:16 / roundup_power2_divisions:16: alignment and bucket granularity to reduce small-object fragmentation.
On top of previous optimizations: c=1 TPOT 9.1 ms → 8.8 ms ( -3.3% ). Later combined with jemalloc, simplified to expandable_segments:True,roundup_power2_divisions:16.
5. Host Memory Allocator: jemalloc Replaces glibc malloc
vLLM host side generates many small objects (tensor metadata, scheduling structs, comm descriptors). glibc ptmalloc suffers arena lock contention and fragmentation under multi-threaded high-frequency allocation. Switch via: export LD_PRELOAD=/usr/lib64/libjemalloc.so.2 Gain scales with concurrency: c=1 -0.1 ms, c=4 -0.3 ms, c=8 -0.6 ms. Higher concurrency → more thread contention → jemalloc thread-cache advantage grows. Zero-risk universal optimization, recommend enabling by default.
6. Operator Cache & Batch Memcpy
ACLNN_CACHE_LIMIT=100000: raises host-side aclnn operator info cache limit, prevents cache eviction during decode causing re-parsing. p99 9.1 ms → 8.9 ms, small but stable, helps eliminate tail-latency spikes. VLLM_ASCEND_ENABLE_BATCH_MEMCPY=1: merges scattered small H2D/D2H copies into batched copies, reduces copy count and launch overhead.
7. MoE Weight Prefetch
At c=1, MoE GroupedMatmul is bandwidth-bound: expert weights far larger than activations, each step reads from HBM. Weight prefetch uses NPU prefetch to pull next-layer weights into L2 during current-layer compute.
Enable via additional-config:
--additional-config '{..., "weight_prefetch_config": {"enabled": true}, "pa_shape_list": [1,4,8]}'Three critical details:
Prefetch threshold change (source): Upstream vllm_ascend/ops/weight_prefetch.py has MOE_PREFETCH_TOKEN_THRESHOLD = 96 (skip prefetch if batch tokens < 96). Online decode batch = concurrency (1–8), never reaches threshold. Changed to 1 to enable prefetch at c=1.
pa_shape_list: Explicitly declare decode batch shapes [1,4,8] so paged attention and prefetch logic handle these low-concurrency shapes.
Shared expert prefetch extension (source): Original only prefetched routed expert w13 (gate_up) weights. Added prefetch of next-layer shared expert gate_up weights in preprocess ( _get_moe_shared_expert_gate_up_prefetch_weight), overlapping with current-layer routed expert GroupedMatmul to fill pipeline bubbles.
Benefits: c=8 TTFT improved (prefill expert weights pre-warmed in L2), c=4 TPOT -0.2 ms, c=1 TPOT -0.1 ms.
8. HCCL Communication Params & CPU Binding
HCCL_BUFFSIZE=128: comm buffer size (MB), tune per message size. HCCL_DETERMINISTIC=false: disable deterministic reduction (fixed chunk/order) for speed; bitwise reproducibility not required for online serving. HCCL_WHITELIST_DISABLE=1: disable whitelist check, reduce connection setup overhead. COMBINED_ENABLE=1: enable operator combined flag (no standalone gain on 35B-A3B but kept for combo). OMP_NUM_THREADS=1 / GE_USE_STATIC_MEMORY=1: limit OpenMP threads to avoid host contention; GE uses static memory to reduce graph engine dynamic allocation. enable_cpu_binding: true (additional-config): bind worker processes to cores. Distinct from aggressive CPU_AFFINITY_CONF=2 which proved negative (see 4.10).
9. CUDAGraph Config: Mode, Capture Sizes, Warmup
cudagraph_mode = FULL_DECODE_ONLY: capture only pure decode steps; prefill and mixed batches run eager for flexibility. cudagraph_capture_sizes: non-speculative use [1,2,4,8]; with MTP ( num_speculative_tokens=4), actual tokens per step = concurrency/DP × (1+spec_len). Example: c=4, TP=2 → 2×5=10, plus bonus tokens → capture sizes adjusted to [5,20,40] (detailed in Part 2). This fixed MTP c=4/c=8 regression from 10.4 ms to 42 ms. cudagraph_num_of_warmups (source: vllm_ascend/platform.py): increased from 1 to 3. Single warmup left residual host-side compilation/init for some aclnn ops (especially MoE fused experts); 3 warmups significantly improved steady-state hit rate, low-concurrency gain ~ -0.2 to -0.3 ms.
10. Failed / Negative Attempts (Pitfall List)
All tested on Qwen3.5-35B-A3B, TP=2, vLLM-Ascend 0.23.0. Key observations:
Ascend has hundreds of env vars; many tightly coupled to SoC model, CANN version, model architecture. Documented "recommended values" cannot replace empirical testing.
Some features (MatmulAllReduce, FlashComm) lack full operator library support on 910C (ascend910_93).
Same optimization can have opposite effects on different models (e.g., FINE_GRAIN_REPLAY_SYNC).
Establish a "must benchmark per model" process rather than memorizing any specific parameter.
This concludes Part 1 of "Domestic Compute: Ascend 910C LLM Inference Optimization", focusing on image upgrades and parameter tuning. Part 2 will cover vLLM-Ascend source modifications, operator-level fusion, and MTP speculative decoding.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
