Ascend 910C LLM Inference Optimization Part 2: Source Code & Operator Tweaks Boost Speed 30%
The article details vllm-ascend source code modifications (MoE shared expert multistream overlap, decode host overhead reduction, block table CPU fast path, GDN linear attention optimizations, sampling/speculative decoding improvements), operator-level optimizations (FUSED_MC2, SuperKernel, fused matmul_allreduce_add_rmsnorm_910c, static kernel compilation fixes), and MTP speculative decoding, achieving 30% TPOT reduction on Qwen3.5-35B-A3B.
Introduction
This is the second part of the Ascend 910C large model inference optimization series. The first part covered hardware/software environment, performance baselines, bottleneck analysis, and parameter tuning without source changes. This part dives into vllm-ascend source code modifications, operator-level optimizations, and MTP speculative decoding, providing complete before/after performance comparisons and methodology summary.
Optimization Practice (II): vllm-ascend Source Code Modifications
1. MoE Shared Expert Multistream Overlap (multistream_overlap_shared_expert)
Motivation: Standard MoE execution order: routed experts (GroupedMatmul) → shared expert → AllReduce, all serial. Two issues found: (1) shared expert computation has no data dependency on routed experts, can run in parallel; (2) under MC2 communication mode, maybe_all_reduce_tensor_model_parallel is no-op for MC2, so shared expert's AllReduce is the only communication overhead and can overlap with computation.
Approach: Move shared expert to independent side stream (shared_experts_calculation_stream) parallel to default stream's routed expert; keep shared expert's AllReduce on side stream; default stream waits for side stream completion.
Implementation (vllm_ascend/ops/fused_moe/fused_moe_0_23_0.py): Changes concentrated in _forward_shared_experts: use npu_stream_switch to enter side stream; after shared expert computation, perform AllReduce on side stream instead of switching back to default stream.
overlap_enabled = self.multistream_overlap_shared_expert and not get_forward_context().capturing
with npu_stream_switch(shared_experts_calculation_stream(), enabled=overlap_enabled):
shared_out = ... # shared expert computation (gate_up matmul + down matmul)
if need_all_reduce: # moe_comm_type in {ALLTOALL, MC2, FUSED_MC2} and not shared-expert-DP
with torch.npu.stream(shared_experts_calculation_stream()):
tensor_model_parallel_all_reduce(shared_out) # AR stays on side stream
torch.npu.current_stream().wait_stream(shared_experts_calculation_stream())Under bf16 non-quantized path, shared expert's part1 (gate_up matmul) and part2 (down matmul) are independent of routed expert output, making parallelism safe.
Reducing Event Overhead
After parallelization, routed side's dispatch/combine/gmm2 events for syncing with shared side become unnecessary. Retaining them adds per-layer overhead. Added forward context flag skip_dispatch_combine_events: set before overlap path entry, reset after exit. Modified moe_comm_method.py (before_dispatch_evt, before_combine_evt set to None), moe_mlp.py (three before_gmm2_evt set to None), ascend_forward_context.py (added flag to extra_attrs whitelist).
Coexistence with FUSED_MC2 and Gains
Upstream had mutual exclusion check: VLLM_ASCEND_ENABLE_FUSED_MC2=1 forced multistream_overlap_shared_expert=False. Analysis confirmed no conflict (overlap handles compute stream parallelism, FUSED_MC2 handles communication operator fusion). Removed restriction, enabling both optimizations to stack.
Gain summary (stacked on prior optimizations): Initial multistream parallel optimization +0.3ms (c=1); skipping redundant events + shared expert weight prefetch added +0.2ms; final non-speculative c=1 TPOT reduced from 8.2ms to 8.0ms.
2. Optimizing Decode Host Overhead
At c=1 decode, batch structure barely changes each step, but upstream model_runner unconditionally executes many copy_to_gpu/fill_/commit ops per step. These non-blocking copies still insert extra host→device DMA, lengthening async pipeline. In vllm_ascend/worker/model_runner_v1.py introduced "previous round state" tracking to skip when structure unchanged:
Lazy-init tracking variables _prev_num_reqs/prev_total_tokens/prev_bt_checksum.
Block table commit only when num_reqs==1 and same as previous step; use block_table.np[0].sum() checksum to detect change, skip commit_block_table() if unchanged.
Tail zeroing (seq_lens, query_start_loc, optimistic_seq_lens fill) only when request count shrinks.
discard_request_indices/mask, req_indices, query_pos, num_scheduled_tokens copy_to_gpu conditional on actual discard/num_reqs change/total token change.
Positions computation rewritten: originally GPU index_select + cast + add three kernels; now use CPU-side precomputed pinned buffer, single H2D copy (self.positions[:n].copy(self._positions_cpu_buf[:n], non_blocking=True)).
Accompanying copy_snapshot_to_gpu pinned staging cache refactor (vllm_ascend/worker/utils.py): upstream clone().pin_memory() reallocates pinned memory each call; changed to reuse preallocated pinned staging per buffer, only copy data not reallocate memory.
3. Block Table CPU Fast Path
Slot mapping (logical token position → KV cache physical slot) default uses NPU Triton kernel _compute_slot_mapping_kernel. Profiling shows fixed ~131us launch overhead on Ascend. In decode, each iteration processes few tokens (small batch), so kernel launch latency far exceeds compute time. Added CPU fast path in vllm_ascend/worker/block_table.py compute_slot_mapping(): when context parallelism not enabled and CPU positions + query_start_loc provided and num_tokens <= 80, compute mapping directly on host using numpy, then single non-blocking H2D write back.
req_idx = np.repeat(np.arange(num_reqs, dtype=np.int32),
query_start_loc_cpu[1:] - query_start_loc_cpu[:-1])
pos_arr = positions_cpu[:num_tokens].astype(np.int64)
pos_block = pos_arr // self.block_size
pos_offset = pos_arr % self.block_size
block_ids = self.block_table.np[req_idx, pos_block].astype(np.int64)
slots = (block_ids * self.block_size + pos_offset).astype(np.int32)
self.slot_mapping.np[:num_tokens] = slots
self.slot_mapping.copy_to_gpu(num_tokens) # single non-blocking H2Dc=1~8 decode num_tokens always <80, so fast path hits 100% in target scenario, saving ~131us kernel launch + sync per step. commit_block_table also switched to reuse pinned staging (_bt_pinned), avoiding repeated pinned memory allocation per step.
4. GDN (Linear Attention) Path Specialized Optimizations
Qwen3.5's 30-layer GDN is decode timeline's most operator-dense region. Original: causal_conv1d → mixed_qkv split (rearrange) → l2norm → recurrent kernel; many small kernels per layer, heavy host overhead and HBM round-trips. Optimized from three angles:
L2Norm Fused into Recurrent Kernel (AscendC Operator Refactor)
Decode path calls triton l2norm_fwd kernel per layer for q and k (2 launches/layer). Fused L2Norm directly into npu_recurrent_gated_delta_rule AscendC kernel: q/k/v normalized on-chip before state update, avoiding multiple HBM round-trips and kernel launches.
csrc changes (csrc/attention/recurrent_gated_delta_rule/):
op_kernel/recurrent_gated_delta_rule_tiling_data.h: tiling struct added useL2Norm field.
op_kernel/recurrent_gated_delta_rule.h: constructor reads useL2Norm, inserts inline L2 norm calc after Cast, before Mul in CopyInQKV phase.
op_host/recurrent_gated_delta_rule_tiling.cpp: writes useL2Norm=1 to tiling.
Python side vllm_ascend/ops/gdn.py: removed decode path's two l2norm_fwd(query/key) calls.
Build: After C++ compile, run python3 setup.py build_ext --inplace to regenerate .so. Requires VLLM_ASCEND_KEEP_MAMBA_MODE=1. Gain: c=1 TPOT 8.9→8.5ms (-0.4ms), c=4 11.7→11.5ms (-0.2ms), c=8 saw 0.5ms regression.
Host-side Micro-optimizations
Conv weight transpose cache: causal_conv1d transposes conv weight each layer each step via transpose(0,1).contiguous(). Cached transposed result keyed by weight data_ptr (conv_weights_T), eliminating ~4.3us × 30 layers ≈ 129us/step.
Rearrange view-based: Original used torch.cat to concat/rearrange mixed_qkv, producing Concat op per layer. Added rearrange_mixed_qkv: at c=1 decode, split slices are contiguous in last dim, directly reshape to view, saving ~123us/step per layer.
Mamba State Copy-Compute Overlap (Multistream Parallelism)
GDN layer's ssm_state (mamba state) requires block copy on request switch/eviction. Original executed synchronously on critical path. Moved do_mamba_copy_block to global_stream, overlapping with update_cos_sin / index_select. Used preallocated torch.npu.Event for sync — event objects preallocated/reused, avoiding ~10us host overhead per step for event create/destroy.
5. Sampling & Speculative Decoding Path Optimizations (Largest TPOT Reduction)
This group reduced c=1 TPOT from 7.02ms directly to 5.12ms (-27%), further to 5.0ms. Core idea: greedy sampling only needs argmax, not full vocab logits/probs computation/storage.
MinTokens EOS Mask Cache (rejection_sampler)
Problem: Upstream MinTokensLogitsProcessor.apply_with_spec_decode rebuilds (rows, toks) mask on host via numpy each decode step, then two torch.from_numpy(...).to(device, non_blocking=True) copies. Source memory is pageable (non-pinned), so H2D copies are actually synchronous — host must wait for device to drain and copy complete, breaking async pipeline, consuming ~30% of step time.
Solution: Mask is constant in "steady state" — as long as each request's remaining min length remaining = min_tokens - len(output) ≥ num_draft_tokens, mask content unchanged. Implemented _apply_min_tokens_spec_cached in vllm_ascend/sample/rejection_sampler.py:
_min_tokens_spec_steady_structure: checks steady state (returns None near min_tokens tail or batch index expired, falls back to upstream per-position logic, semantics identical).
get_min_tokens_spec_mask_indices: precomputes mask as device int64 tensors (rows_t, toks_t, bonus_rows_t, bonus_toks_t) cached by (batch structure, request structure), rebuilds only on structure change (rebuild uses pinned memory async copy).
Per-step application: single pure-device logits.index_put((rows_t, toks_t), neg_inf_tensor), no numpy rebuild, no H2D copy, no sync.
Key detail: Ascend's masked_select compiles to NonZero operator triggering host-device sync; all mask applications changed to pure integer advanced indexing. This optimization alone contributed -1.9ms/token (7.02→5.12ms), largest single gain among source optimizations.
Vocab-Parallel argmax: Skip Full-Vocab All-Gather
argmax finds input index maximizing function, not max value itself.
Problem: Upstream compute_logits does TP all-gather on full vocab logits, communication O(batch × vocab_size). Greedy sampling only needs per-row argmax.
Solution: Local argmax per TP shard, only aggregate (max_value, max_index) pairs, communication reduced to O(batch × 2 × tp_size). With 150k vocab, TP=2, communication volume drops ~31,000x. Implemented in vllm_ascend/spec_decode/step3p5.py _draft_greedy_shard_argmax:
logits = lm_head.quant_method.apply(lm_head, hidden_states, bias=None) # only local shard
logits[..., -num_pad:] = -float("inf") # mask vocab padding in shard
local_max_vals, local_max_indices = logits.max(dim=-1)
global_indices = local_max_indices + vocab_start # shard index -> global vocab index
local_pair = torch.stack([local_max_vals.float(), global_indices.float()], dim=-1)
gathered = tensor_model_parallel_all_gather(local_pair, dim=-1) # gather (value,index) pairs
max_rank_idx = gathered[:, :, 0].argmax(dim=-1, keepdim=True) # pick shard with max value
top_tokens = gathered[:, :, 1].gather(dim=-1, index=max_rank_idx) # get corresponding token idCorrectness argument: soft_cap (tanh) and scale are monotonic transforms, don't affect argmax; padding positions set to -inf; "shard argmax + cross-shard compare" strictly equivalent to "full vocab argmax".
Deployed in three places: draft side (MTP draft generation) _draft_greedy_shard_argmax; target verification side (spec-decode verify) _spec_greedy_shard_argmax in model_runner_v1.py with local shard EOS mask (reusing MinTokens mask cache indices, filtered to shard at host build time); normal decode side _vocab_parallel_argmax_eos_masked in model_runner_v1.py, pre-sampling at execute_model stage (ExecuteModelState.pre_sampled_tokens), sample stage directly constructs SamplerOutput, bypassing full vocab materialization and communication.
Greedy Sampling Fast Path: Skip bf16→fp32 Conversion
General sampling path converts bf16 logits to fp32 entirely. For pure greedy sampling, this can be safely skipped: bf16 exponent wider than fp16, any bf16 value exactly representable in fp32, so argmax(bf16) == argmax(bf16.to(fp32)), elementwise comparison order unchanged. vllm_ascend/sample/sampler.py AscendSampler.forward adds fast path: when all_greedy, no logprobs, no penalties, no allowed-token/bad-words constraints, and logits processors don't change argmax, directly argmax on bf16 logits, saving one full-vocab (~248K elements/step) cast kernel.
Pre-sampled Tokens Fed Directly to Rejection Sampling
Vocab-parallel argmax produced token ids fed directly to rejection sampling: rejection_sampler.py added forward_greedy_presampled, receives full greedy batch pre-sampled argmax ids (target draft rows + one bonus row per request), runs triton/pytorch rejection_greedy_sample for verification, never needing full vocab logits or fp32 copies.
6. Other Source-Level Micro-optimizations
Individually minor but decode timeline accumulates "few microseconds per layer per step":
RMSNorm (1.0+weight) cache: Gemma-style RMSNorm does 1.0 + self.weight + .contiguous() each forward. Weights static during inference; cache result keyed by id(self) (layernorm.py GEMMA_WEIGHT_CACHE, patch_qwen3_5.py get_weight_plus_one), saving 40 layers × 2 sites add+copy per step.
Rotary embedding expression merge: RoPE injects position info by rotating Query/Key vectors. update_cos_sin prepares cos/sin tensors per token position from precomputed cache. Since cos/sin numerically symmetric with identical shape/transform flow, merged optimization: combine cos/sin into single tensor, do one index_select + shape transform, then chunk back to cos/sin, eliminating half redundant ops.
Remove redundant copy: rope_dsv4.py ComplexExpRotaryEmbedding.forward originally copied result to preallocated buffer then returned; changed to direct return x.view(ori_shape) view.
Attention weight dual-stream prefetch: Weight prefetch originally only covered MoE weights. Leveraged "layers built sequentially" property to create layer registry: when layer L enters its MoE (longest compute + AR window), prefetch stream loads layer L+1 attention projection weights (qkv_proj/o_proj etc.); layer L+1 waits on prefetch stream before consuming, hiding attention weight HBM load inside previous layer's MoE window (patch_qwen3_5.py, VLLM_ASCEND_ATTN_WEIGHT_PREFETCH enabled by default).
Optimization Practice (III): Operator-Level Optimizations & Compilation Fusion
Decode critical path spends much time in fixed sequences "matmul → AllReduce → residual add → RMSNorm" and MoE communication-compute interleaving. Compressed these via MC2 communication fusion, SuperKernel, and custom 910C fused operators.
FUSED_MC2: MoE Communication-Compute Fusion
MC2 (Mixture-of-experts Communication & Computation) is Ascend's fused mechanism for MoE dispatch/combine, interleaving AllToAll communication with GroupedMatmul inside operator. vllm-ascend provides two-level switch via env: export VLLM_ASCEND_ENABLE_FUSED_MC2=1 # enable fused compute-communication operator. Level=1 shows stable gain; level=2 (more aggressive fusion) no extra gain on 35B-A3B. Also relaxed MC2 path token capacity limit in ascend_forward_context.py from min(num_tokens_per_tp_rank, 512) to 1024, allowing larger batches to use MC2 RS+AG path instead of ALLTOALL, leaving headroom for future capture size expansion.
SuperKernel: Merge Small Kernels into Large Kernel for Joint Loading
Low concurrency: single kernel compute often only few microseconds, kernel load/launch overhead dominates. SuperKernel merges multiple consecutive small kernels at compile time into one large SuperKernel, synchronous launch, reducing kernel load count and communication overhead, naturally forming inter-operator pipeline overlap. Enabled via compile options in vllm-ascend (migrated, enabled by default with static kernel, see compiler_interface.py):
options["static_kernel_compile"] = True
options["super_kernel_optimize"] = True
options["_vllm_aclnn_static_kernel_sym_range"] = \
_compute_decode_cudagraph_batch_sizes(vllm_config) # limit static compile batch sizessym_range restricts static kernel compilation to cudagraph-captured batch sizes, avoiding compile time explosion. With SuperKernel enabled (stacked on all prior optimizations), non-speculative c=1 TPOT 8.5→8.2ms (-0.3ms); MTP enabled gain further amplified (see "MTP Speculative Decoding" section). Compiler hash factor includes super_kernel_optimize, ensuring cache invalidation on toggle.
Fused Operator: matmul_allreduce_add_rmsnorm_910c
Motivation
Enabling VLLM_ASCEND_ENABLE_MATMUL_ALLREDUCE for compute-communication fusion on 910C threw "SoC version ascend910_93 verification failed" — operator library didn't cover that SoC. Developed custom fused operator for 910C dual-die TP=2 topology, compressing decoder layer's heaviest sequence:
o_proj(matmul) + AllReduce + residual add + RMSNorm(1+w)
into single operator. Python wrapper in vllm_ascend/ops/fused_ar_norm.py, controlled by VLLM_ASCEND_FUSED_O_PROJ_AR=1, integrated in patch_qwen3_5.py AscendQwen3_5DecoderLayer.forward: attention module passes residual and (1+w) norm weight to fused operator after attn_output, entire epilogue done in one shot, layer forward only receives fused output as new residual.
Operator Architecture: AIC/AIV Dual-Core Pipeline
Operator source at csrc/mc2/matmul_allreduce_add_rmsnorm910c/, uses AscendC AIC (Cube) + AIV (Vector) dual-core cooperative design:
AIC side: Catlass-based fine-grained matmul (L1 tile 16×224×512, double buffering). After each (mTile, nTile) output tile, sets corresponding tile-ready flag in GM (global memory).
AIV side: Polls AIC tile flags (WaitForAicTile); when tile ready, immediately launches cross-rank fine-grained reduction — two dies directly read/write each other's DataBank/StatusBank via HCCS (RANK_DATA_STRIDE=1MB layout), completes reduce-add then RMSNorm (each computes sum-of-squares, cross-rank aggregates reciprocal, normalizes output), all without returning to host.
Pipeline protocol: GM flags divided into three segments: AIC tile ready flags, cross-rank protocol state (protocol/add ready), RMS ready. As AIC progresses, AIV's reduction and normalization increasingly overlap, forming fine-grained matmul-communication pipeline.
Shape constraints & reliability design: Operator supports M ≤ 40 rows (covers MTP bs8 × (4+1)=40 verification scenario), TP=2, bf16, 2048 dim (Qwen3.5-35B-A3B hidden size). Python side first guard (unsupported shapes return None, fall back to unfused path). Operator internal second guard — unsupported shapes fall back via c10::Dispatcher to standard operator combo (matmul + vllm::all_reduce + add_rms_norm, provided by new header csrc/mc2/call_vllm_tensor_op.h). Dual fallback guarantees correctness never depends on fused operator's shape coverage.
Gains
After integration (stacked on all prior optimizations): c=1 TPOT 5.2→5.1ms, c=4 9.2→9.0ms, c=8 12.9→12.3ms (p99 18.8→14.8ms); TTFT c=1 213→179ms (-16%), c=8 432→307ms (-29%) because prefill also uses fused operator.
Static Kernel Compilation Fix: 37/43 → 43/43
Static kernel precompilation (see part 1) requires successful compilation of all graph operators. Compiling 43 custom operators initially only 37 succeeded. Two issues found:
Hardcoded header paths: Build script referenced /tmp/vllm-ascend/.../cann_compat.h absolute path, invalid after container restart. Fixed to relative path based on PYF_PATH, recovered to 42/43.
CausalConv1d empty input handling: When bias/initialStateMode/numAcceptedTokens params are None, tiling descriptor positions misaligned. Fixed using "nullptr" empty tensor descriptor + stripping stride/offset/is_view attributes, achieving 43/43 (100%) compilation success.
Direct benefit: cudagraph capture covers all decode shapes, decode throughput up ~4.2%.
Static Kernel & MTP Compatibility
Enabling static kernel + MTP speculative decoding simultaneously caused immediate inference request error — draft (MTP draft) model's static compilation triggered AICore out-of-bounds during cudagraph capture. Root cause: MTP draft model's FX graph differs from target (its fc layer consumes "token embedding concatenated with target hidden states" 2×hidden_size tensor), this graph compiled to static kernel goes out-of-bounds. Fix: compiler_interface includes super_kernel_optimize (static kernel + super kernel synergy), under which MTP graph compiles normally (cudagraph capture 5/5 passes).
cudagraph Capture Sizes Must Match Post-Speculative Token Counts
Initial MTP enable caused severe regression: c=4 TPOT 10.4→42.2ms, c=8 13.5→42.9ms. Cause: after speculative decoding, each decode step actual token count = concurrency/DP × (1 + num_speculative_tokens) + possible bonus. E.g., c=4/TP=2 ≈ 10-20 tokens, but default capture sizes [1,2,4,5,8] completely miss these shapes, causing speculative steps to fall outside graphs.
Fix: reset capture sizes via formula:
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY",
"cudagraph_capture_sizes": [5, 20, 40]}'After fix: c=4 TPOT 42.2→10.4ms, c=8 42.9→13.5ms.
Before/After Performance Comparison
c=1 Optimization Waterfall: Each Step's Contribution
Table summarizes Qwen3.5-35B-A3B at c=1, input 1024/output 512, no speculative decoding, sequential TPOT evolution (752t single-card dual-die, TP=2).
Largest three single items: version upgrade (-15.4%), static kernel (-18.7%), HCCL AIV (-6.6%), together contributing ~70% of total reduction; remaining dozen items contribute ~30%.
Overview I: Without Speculative Decoding (random → real dataset)
After stacking all non-speculative optimizations (910C-560t, single-card dual-die, TP=2), comparison with baseline and H20.
Overview II: With MTP Speculative Decoding (after MTP-specific source optimizations)
On top of 8.2 baseline, enable MTP (num_speculative_tokens=4) + SuperKernel.
Overview III: Real Business RL-Finetuned Model + Real Business Dataset (Final State)
Final state: real business dataset c=1 TPOT reaches 5.0ms, 64.8% reduction from baseline. Production validation used real-business RL (GRPO/PPO, based on verl) finetuned Qwen3.5-35B-A3B model with three real dataset stress tests. After stacking final batch of source optimizations (min_tokens mask cache, vocab-parallel argmax, lm_head sharded compute):
On gsm8k public dataset (math reasoning, token distribution differs from real business), optimization effect even more pronounced:
Difference highlights: speculative decoding gain strongly correlates with acceptance rate, which is data-distribution dependent. Evaluating domestic compute substitution should use real business datasets; public datasets only for reference.
Generalization Verification: Qwen3.6-27B
Verified optimization generality on same env (910C-560t, v0.23.0 internal branch) for dense model Qwen3.6-27B. Note: this model has compatibility limits with MTP + static kernel combo (mixed use errors on inference request, need disable enable_super_kernel or static_kernel; resolved in later 0.26.0rc image).
27B dense model (non-MoE, non-mixed-attention) also benefits.
TTFT & Other Metrics
TTFT: Baseline Ascend already beat H20. Source optimizations further improve: fused operator brings c=1 TTFT 213→179ms (-16%), c=8 432→307ms (-29%).
End-to-end latency: 1024-512 request latency 5.09s→4.04s (-21%).
High-concurrency throughput: Static compilation fix (43/43) boosts concurrency-10 decode throughput 4.2%. After cudagraph capture range expanded from 49 to 70, c=10 throughput 173→477 tok/s — shows low-concurrency methodology alone insufficient; high concurrency needs separate capture range & scheduling tuning.
Methodology Summary & Outlook
Methodology Summary
Profiling: All optimizations originated from msprof timeline concrete bottlenecks (operator launch overhead, host gaps, sync copies). Each optimization did single-variable A/B testing with records; failed attempts also recorded.
Low-concurrency optimization = reduce host overhead: At c=1, device operators in microseconds; TPOT bulk is host scheduling, kernel launch, H2D copies, sync waits. Static kernel, SuperKernel, CPU fast path, mask cache, redundant copy skip — all compress host-side microseconds.
Communication optimization: Vocab-parallel argmax cuts communication 30,000x; fuse_allreduce_rms, MC2, 910C fused operator push communication into compute pipeline. Multistream overlap lets shared expert communication run concurrent with routed compute.
Speculative decoding: MTP gives clear TPOT boost; do baseline optimization first, then enable speculative decoding.
Version upgrades: After version upgrade, all prior source mods based on old version must be re-migrated and validated. Used "migrate baseline optimizations first, then stack incrementally" to complete migration from 0.23.0rc2 to 0.23.0 release.
Gaps & Future Work
High-concurrency direction: This round focused on online low-concurrency (c≤8). High-concurrency scheduling, memory, throughput optimization (e.g., PD disaggregation, larger capture range, DP attention) next phase focus.
Compilation optimization: Explore PGO (Profile-Guided Optimization), more aggressive operator fusion (e.g., QKV proj + RoPE fusion).
Upstream collaboration: Contribute generally-useful optimizations (min_tokens mask cache, vocab-parallel argmax, block_table CPU fast path, etc.) back to vllm-ascend community, reducing long-term maintenance cost.
Scale expansion: Replicate same methodology & best-practice scripts to more machine types (single-node multi-card, multi-node) and more models (Qwen3.5-9B/27B, future dense large models), accumulating domestic compute inference optimization experience.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
