SGLang Multi-Hardware Plugin Architecture and Kunlun Chip Adaptation: A 2-Hour Model Upgrade Case Study

SGLang Multi-Hardware Plugin Mechanism

SGLang's codebase exceeds one million lines and supports CUDA, ROCm, NPU, XPU, MUSA, and other backends. As backends proliferate, hardware-specific implementations for attention, KV pool, graph runner, quantization, and communication scatter across the main repository, increasing maintenance burden and feature velocity drag. The community adopted a Device Plugin architecture (similar to PyTorch's approach) to extract hardware-specific code into separate plugins, leaving only generic paths in the main repo.

SGLang-Kunlun Plugin Architecture

The SGLang-Kunlun plugin (open-sourced at https://github.com/baidu-baige/sglang-kunlun) implements two core mechanisms:

Platform : Each hardware platform registers its own communication library (XCCL for Kunlun) and operator backend. Attention backends differ per platform; Kunlun provides KunlunAttentionBackend. The main framework leaves hooks (e.g., get_xxx_cls) for early instantiation of platform-specific attention objects, ensuring registered operators and layers remain reusable across upstream iterations.

Hook : Fine-grained platform patches (Triton compiler workarounds, communication library initialization data sizing) are unified via an official decorator-based API, replacing scattered monkey patches.

Two Key Design Decisions

cuda-like masquerade : Internally the device is named "kunlun", but when SGLang queries the device type, the plugin responds "cuda". This allows wholesale reuse of upstream CUDA-written logic (graph runner, etc.) without massive rewrites.

Python entry points for plug-and-play : Users run pip install sglang then pip install sglang-kunlun. The plugin auto-detects Kunlun hardware via torch_xmlir presence; only then does it register the Kunlun Platform. Activation swaps implementations via hooks. Multiple plugins coexist; SGLANG_PLATFORM selects one. Zero main-repo changes required.

Five-Step Standardized Adaptation Process

Each step has hard acceptance criteria, enabling rapid, reliable tracking of upstream changes:

Requirement Alignment : Read main-repo feature implementation, locate Kunlun backend touchpoints, decide Platform factory vs. hooks. Output: precise landing-point judgment.

Interface Adaptation : Add factory methods and default parameters on KunlunSRTPlatform; add hooks where needed. Acceptance: service starts and emits tokens.

Operator Completion : Replace torch/CUDA/Triton implementations with Kunlun operators. Acceptance: no Triton compilation failures, no fallbacks.

Precision Alignment : Match module and operator outputs against golden references. Acceptance: GSM8K, LongBench scores match baselines.

Performance Optimization : Profile to locate bottlenecks; apply fused operators, graph capture, or parallel strategy tuning. Measure throughput and TTFT.

This process enabled a 2-hour upgrade of DeepSeek V4 Flash from v0.5.17 to v0.5.19 .

Supported Models and Capabilities

Quantization: W8A8 (throughput-optimized) and W4A8 (precision-optimized). DeepSeek V4 Flash uses W8A8 with TP+CP+EP prefill, DPA+TP+EP decode, and DSpark speculative decoding. DeepSeek V4 Pro uses W4A8. Also supported: MiMO-V2-Flash, MiniMax-M3, DeepSeek V3.2, GLM5.x. Capabilities include PD disaggregation, full parallel suite (CP/DP/PP/TP/EP/DPA), attention variants (MHA/GQA/SWA/MLA/NSA/DSA), speculative decoding (EAGLE/DSpark), compress-tensor W8A8/W4A16, and HiCache L1/L2/L3 multi-level KV cache.

Performance Optimization Case Studies

Operator Optimization: HCA/CSA Dual-Stream Execution

DeepSeek V4's HCA (continuous KV cache, aka C128) and CSA (sparse KV cache, aka C4) cover all data scenarios. At context length 60K, M=16384:

HCA Flash: MFU 31.8%, MBU 59.8%; Pro: MFU 45.5%, MBU 50.3%

CSA Flash: MFU 31.3%, MBU 58.2%; Pro: MFU 48.6%, MBU 46.0%

CSA leverages Kunlun Gen3's hardware feature: vector and matrix compute clusters run simultaneously. Vector cluster moves sparse KV into L3 ring buffer while matrix cluster computes attention, overlapping memory access and compute. Dual-stream yields 1.55x speedup on Flash, 1.38x on Pro , applicable to any sparse attention.

CPU Dispatch Overhead: CUDA Graph Capture

Small-batch decode throughput is bound by per-operator CPU dispatch. Four actions eliminate this:

Piecewise capture: only static subgraphs captured; dynamic parts fall back to eager.

Full capture: entire static intervals captured, removing remaining dispatch overhead.

Sync optimization: reduce explicit syncs inside/outside graphs.

Communication operators folded into graph to avoid graph breaks.

Since Kunlun's device backend is CUDA-compatible, all CUDA-side optimizations transfer directly.

Communication-Computation Overlap: Dual-Batch Overlap with DeepEP

Requests split into two micro-batches interleaved: while one performs dispatch/combine all-to-all, the other computes attention and MoE. Communication latency hides inside computation, eliminating serial bottleneck in expert parallelism. Implementation mirrors CUDA community approach.

AI-Assisted Adaptation: Plugin + Harness

Adapting a feature in the main repo touches 40-50 files with 1-2 line changes each — high cognitive load. The plugin isolates adaptation surface. Directly prompting AI on the main repo fails: single-point patches misalign, correctness/performance/fallback/multi-process concerns are missed, and patches don't persist across upgrades.

Solution: converge adaptation expertise into a focused Skill , driven by a Harness that executes the five-step loop with AI handling repetitive retrieval, drafting, batch verification, and logging, while engineers retain architectural decisions and final sign-off.

Requirement Alignment: AI compares main-repo features, Platform methods, Kunlun adaptation points; generates localization report.

Interface Adaptation: AI generates factory methods/defaults from Platform Map; hooks only where targets are explicit.

Operator Completion: AI selects Kunlun implementations per Operator Recipe; logs fallbacks and compilation failures.

Precision Alignment: AI orchestrates module-vs-golden comparison and GSM8K/LongBench baselines.

Performance Optimization: AI summarizes profiling, candidate optimizations, regression results; engineer picks optimization combo.

Each step has AI tasks, human gates, and verifiable artifacts.

Real-World Agent Diagnostics

PP Pipeline Bubble Analysis (DeepSeek V4 Pro, 60K prefill, TP8×PP4)

Agent auto-deploys, runs profiling, searches layer-balancing candidates from per-stage traces, proposes 16-15-16-14 split. Analyzes PP bubble root cause via wait chains: PP1 waits PP0, PP2 waits PP1, PP3 waits PP2; hypothesizes PP0 host submission gap. All judgments autonomous; report gives engineers high-confidence validation without line-by-line reading.

irecv/wait Serialization Bug Fix

Agent traced a performance issue to a single code line: irecv A → wait A → irecv B → wait B serialized what should be parallel receives. Fix: post all receives first, then unified wait, creating tensor buffers for A and B upfront. End-to-end throughput improved ~10%.

Four-Level Skill Accumulation

Platform Map : factory methods, feature landing points, default parameters, conflict records — reduces code reading and misjudgment.

Feature Diff : main-repo version deltas, affected modules, impacted interfaces, migration checklist — lowers version-tracking cost.

Operator Recipe : Kunlun operator selection, sharding rules, fusion strategies, fallback conditions, failure cases — makes performance optimization reproducible.

Validation Runbook : unified precision benchmarks, performance benchmarks, environment specs, acceptance records — turns "it runs" into "provably correct".

Each completed model/feature updates the Skill; next adaptation starts from a higher baseline. One adaptation becomes the accelerator for the next iteration.

Baidu Baige Loong Open-Source Ecosystem

LoongForge : unified multimodal training framework (LLM, VLM, diffusion, embodied models)

LoongSage : production-grade Agentic RL for frontier models

LoongFlow : Harness framework that learns from its own runs

LU-KV : long-horizon KV cache utility optimization

Covers training, RL, Harness, and KV optimization — full stack.