Cut First‑Token Latency by 3.25×: Introducing HYPIC for Position‑Independent Caching in Hybrid‑Attention LLMs

RAG and Agent services construct prompts from dozens to hundreds of independent text fragments, often reaching tens of thousands of tokens; the prefill stage then dominates compute cost and cache misses can make tail latency reach dozens of seconds.

Two research directions have emerged: Position‑Independent Caching (PIC), which reuses KV cache across arbitrary prefixes, and linear attention, which compresses the KV history into a fixed‑size recurrent state. PIC requires token‑wise KV, while linear layers expose only a per‑request compressed state, so the two cannot coexist in hybrid‑attention models.

HYPIC is the first system that brings PIC to hybrid‑attention LLMs. It consists of three tightly coupled mechanisms: (1) for linear layers, cache a segment‑cumulative transition operator T_C and the initial state S(C|0) to enable constant‑time state composition; (2) for full‑attention layers, introduce a "seam window" that recomputes the first w tokens of each reused segment; (3) apply segment parallelism that scatters cold segments to multiple workers and combines results, using longest‑processing‑time‑first (LPT) scheduling and pipeline overlap.

For linear layers, a naive addition of end states works only for plain linear attention; advanced variants (RetNet, Mamba2, GLA, DeltaNet, GDN, KDA) require the transition operator T_C. HYPIC stores the pair (T_C, S(C|0)) during the first prefill; later reuse composes states as T_C2·S(C1)+S(C2) in O(1) time. On Qwen3.5‑35B‑A3B the resulting state differs from full recomputation by only 6×10⁻⁵, within FP16 noise. For convolutional or RoPE variants, HYPIC adds "convolution state warm‑up" and "state rotation" patches to keep alignment.

Full‑attention layers still need PIC. Observation shows KV splice error concentrates at the start of each reused segment, so HYPIC skips caching the first w=8 tokens, recomputes them on reuse, and thereby restores cross‑segment attention with negligible overhead.

Segment parallelism turns long cold requests into parallelizable workloads. A router detects cache‑miss status, dispatches cold segments to multiple scatter workers, and a combine worker assembles the final state. This reduces cold‑prefill cost from O(n·|C|) to O(⌈n/m⌉·|C|+c). LPT balances worker load, and pipeline overlap prevents the combine worker from being stalled by transmission.

Implemented in SGLang (~14 k lines of Python/Triton) on an 8×H20 cluster, HYPIC was evaluated on four production hybrid‑attention models (Ring‑mini/flash‑linear‑2.0, Qwen3.5‑35B‑A3B/122B‑A10B), four public datasets (HotpotQA, TriviaQA, MultiNews, GovReport) and a production RAG trace. Results: average TTFT reduced by 3.25× (model‑wise 2.77‑4.05×) with quality loss of only 1.71 points (Qwen3.5‑35B even +0.47); sustainable QPS increased by 1.66× (1.49‑1.85×) under a 1 s TTFT SLO, and peak single‑card throughput rose 1.3‑1.5×. Segment‑parallel cold requests (32 k tokens) sped up from 2.83 s (single worker) to 0.49 s (8 workers), a 5.7× gain; LPT outperformed round‑robin (3.6× vs 2.4×). One‑time construction of (T_C, S) costs only 5.2‑6.7% of a forward pass, and the cost is amortized over many reuses.

In summary, HYPIC brings position‑independent caching to hybrid‑attention LLMs by (1) caching segment‑cumulative transition operators for linear layers, (2) repairing full‑attention cross‑segment attention with a tiny seam window, and (3) parallelizing cold prefill via segment parallelism. This yields faster, cheaper, and more scalable inference for long‑context RAG/Agent workloads, and future work will explore distributed PIC management and multi‑level cache hierarchies.