Context Parallelism Slashes TTFT by 80% for 128K-Token LLMs

Background and Motivation

As large language models (LLM) require ever longer context windows, token-to-first-token latency (TTFT) and GPU memory become critical bottlenecks, especially for 128K-token inputs such as legal contracts or technical manuals.

Announcement

On 23 December 2025 the SGLang community announced that Baidu Baige’s AIAK team merged a Context Parallelism (CP) implementation for DeepSeek V3.2 into the main SGLang branch. Benchmarks show up to an 80 % reduction in TTFT at a 32K sequence length, approaching second‑level response times.

Open‑source PR: https://github.com/sgl-project/sglang/pull/12065

1. DSA Architecture Challenges and Evolution of Parallel Strategies

Traditional TP + SP

Tensor Parallelism (TP) splits weight matrices along the hidden dimension H, distributing large matrix multiplications across multiple GPUs to lower TTFT.

Sequence Parallelism (SP) splits activations (e.g., KV cache) along the sequence dimension L, preventing out‑of‑memory (OOM) failures for long sequences.

DSA Core Mechanism

DeepSeek Sparse Attention (DSA) replaces the O(L²) quadratic cost of classic attention with an Indexer that selects the top‑K most relevant key tokens for each query token, reducing complexity to O(L·K).

Indexer quickly filters the full sequence to the top‑K keys per query.

Overall attention cost drops from O(L²) to near‑linear O(L·K).

2. Engineering Difficulties of Deploying DSA

Even with sparse indexing, a single GPU cannot handle 128K tokens because:

QKV projection still incurs O(L) work and the Indexer’s relevance search still touches O(L²) data.

Tensor Parallelism on the hidden axis H conflicts with the Indexer’s reduction step, causing expensive AllReduce communication that erodes TP gains.

3. Context Parallelism (CP) Core Principles

CP avoids splitting the hidden axis H and instead partitions the sequence axis L across N ranks. Each rank receives 1/N of the query tokens via the cp_split_tokens module.

This distributes both QKV projection and Indexer workload, lowering per‑GPU computation to O(L²/N). The approach yields near‑linear TTFT reduction.

Load‑Balanced Sequence Splitting

Hidden states are divided into 2N sub‑chunks.

“Head‑tail pairing” re‑orders chunks so each rank processes a balanced amount of work, further reducing TTFT.

4. End‑to‑End Mixed‑Parallel Pipeline

Data flow:

After embedding, cp_split_tokens performs 2N load‑balanced re‑ordering and distributes tokens to each rank.

Within each rank, local TP (size = 1) computes Q_i, K_i, V_i for its 1/N token slice, avoiding AllReduce.

All ranks gather partial K_i and V_i via AllGather to reconstruct full K_full and V_full, then re‑order with rerange to restore correct sequence order.

Sparse attention uses Indexer_prepare and MLA_prepare to compute attention only on the selected top‑K keys.

Expert parallelism (MoE) is integrated via moe_dense_tp1 and Deep_EP to keep CP and MoE efficient.

After 61 layers, hidden_states_allgather_rerange aggregates hidden states, and logits_processor produces the final output.

5. Impact and Deployment

The CP solution has been deployed on Baidu Baige’s AI compute platform and powers the DeepSeek V3.2 long‑text inference service on the Baidu Qianfan large‑model platform. Ongoing open‑source contributions will keep the design available to the broader SGLang community.