DSpine: Depth-Wise Causal Injection for Parallel Speculative Decoding, 4x SGLang Throughput

In lossless LLM inference acceleration, speculative decoding has become standard. Autoregressive draft models like EAGLE-3 predict accurately but require serial forward passes per token. Diffusion-inspired DFlash parallelizes drafting — one backbone forward emits a whole block of candidate tokens — but suffers from a core paradox: later positions compute without knowing which token the previous position will finally choose, causing multimodal collisions and suffix decay.

Recent works (DSpark, Domino, DFlash2) attempt to inject causal dependencies, but either add lightweight serial heads after the backbone (delaying dependency awareness) or mix adjacent hidden states locally (conflating context and prediction signals).

DSpine (Draft in Parallel, Condition Through Depth) addresses this with the principle "simultaneous drafting, layer-by-layer relay." From layer 1, it extracts the predecessor's forming prediction features into a unified token embedding space and injects them into the immediate successor; at the final layer, a Transition Cache precomputes adjacent candidate combination scores for parallel conditional decoding.

Key Empirical Insights

Layer-wise probing and intervention experiments on parallel draft models yielded two findings:

Early prediction signals emerge in shallow layers: At the same shallow depth, earlier block positions show far higher token prediction accuracy than later positions, meaning predecessors develop readable prediction cues before final token selection.

Immediate predecessor is critical and earlier injection helps more: Even with perfect earlier context, corrupting the immediate predecessor drops successor accuracy by 31.8 percentage points on average. Injecting the same reliable predecessor feature at shallow layers improves accuracy by 14.8 percentage points over later injection.

Conclusion: adjacent causal dependency must enter successor representation from layer 1, not as a post-hoc fix.

DSpine Architecture

Four core designs build the causal chain along network depth while preserving single-forward, same-layer parallel updates:

1. Unified Transfer Space

Instead of passing raw hidden states (mixed with context) or doing full vocab decoding mid-network, DSpine uses the frozen target LM Head's output embedding matrix. After normalization and PCA whitening (removing shared components, keeping principal directions), a compact token table is obtained. Each layer projects hidden states to prediction features; at the end, confirmed discrete predecessor tokens are looked up as token vectors. Both share the same geometric space and algebraic form, enabling a single injection pathway for inter-layer feature passing and final conditional decoding.

2. Layer-wise Gated Adjacent Injection

After each Transformer layer, position t receives predecessor t-1's prediction feature, computes a gate with its own feature, decides how much to absorb, and writes back to the residual stream. Parallelism is preserved because within a layer all positions read pre-update features simultaneously; predecessor's layer-l output feeds successor's layer l+1 computation, unfolding the causal chain along depth without extra serial backbone steps.

3. Layer-wise Output Embedding Supervision

To encourage early sharp prediction features aligned with final token embeddings, DSpine applies a directional alignment loss (cosine similarity) between each layer's injected feature and the ground-truth token's compact embedding. Ablation shows that imposing full-vocab cross-entropy (Token CE) at intermediate layers reduces acceptance length below the unsupervised DFlash baseline, because it forces hidden states into premature classification, crowding out context representation. The low-dimensional cosine constraint lets prediction cues surface early while leaving full hidden states free for context modeling.

4. Final-Layer Conditional Decoding with Transition Cache

After backbone computation, each position selects an initial Top-K candidate set. Left-to-right token commitment begins: once a predecessor token is fixed, the last-layer injection module swaps its input from prediction feature to true token vector, correcting successor scores (Last-Write Refinement). To avoid serial matrix multiplications, DSpine precomputes all adjacent candidate combination scores in a Transition Cache — a single parallel pass before decoding. During left-to-right selection, only fast table lookups are needed, eliminating per-position serial matmul overhead.

Experimental Results

Evaluated on Qwen3-4B and Qwen3-8B across 7 benchmarks (GSM8K, MATH-500, HumanEval, MBPP, LiveCodeBench, MT-Bench, Arena-Hard) under greedy (temp 0) and sampling (temp 1).

Acceptance Length: 28/28 First Place

Qwen3-8B: Temp 0 average acceptance length rises from DFlash 3.77, DSpark 4.32, DFlash2 4.39 to 4.82 (GSM8K 5.67, MATH-500 5.49). Temp 1 average from DSpark 3.92 to 4.42 (+12.8%).

Qwen3-4B: Temp 0 and 1 averages reach 4.75 and 4.38 , leading strongest baseline by 10.1% in sampling.

SGLang End-to-End Throughput: 40/40 First Place

Tested on GSM8K, MATH-500, HumanEval, MBPP at concurrency 2–32.

vs DSpark: Qwen3-8B avg + 19.0% (max 22.1%), Qwen3-4B avg + 20.1% (max 22.5%).

vs DFlash/DFlash2: Qwen3-8B avg +23.3% over DFlash, + 11.1% over DFlash2; Qwen3-4B +6.5% over DFlash2.

vs native AR: Qwen3-8B up to 4.01x speedup, Qwen3-4B up to 3.80x .

Latency Breakdown

All 5 injection modules add only 0.15 ms per step (with operator fusion + CUDA Graph).

Transition Cache makes final conditional decoding less than half the latency of DSpark/Domino's serial heads.

Total per-step latency (7.14 ms) matches DFlash2 (7.19 ms), but DSpine's higher acceptance length yields 10.9% lower average decode time per accepted token .

Agent-Oriented Evolution: Qwen3.8-27B

Trained a DSpine draft model for Agent scenarios on Qwen3.8-27B, compared against open-source DFlash2 weights. Benchmarks: SWE-bench Pro, τ²-bench, WorkBuddy Bench (Code/Web/Office) with low/xhigh thinking intensity.

Average acceptance length up to 5.13 , WorkBuddy Office (low) peaks at 5.63 .

Decoding throughput vs DFlash2 avg + 21.2% , vs AR up to 3.84x .

Real-office six-task average acceptance length + 26.1% .

Open-sourced production-ready Qwen3-8B and Qwen3.8-27B draft weights; inference implementation packaged as a lightweight SGLang plugin. DSpine is now serving Tencent WorkBuddy online inference workloads.

Paper: https://arxiv.org/abs/2609.36173 Code: https://github.com/tencent/dspine Weights: https://huggingface.co/collections/tencent/dspine Project:

https://dspine-project.github.io