MapFlow: Cross-Model Attention Map Reuse Cuts LLM Prefill Latency 15-20%

Cross-Model Context Handoff and the Cost of Recomputation

Multi-model collaboration — where a large model handles planning and smaller models handle refinement or summarization — offers a path to balance quality and cost in AI agent applications. However, when a task switches from one model to another, the receiving model typically recomputes the entire shared context (documents, tool descriptions, conversation history) during its prefill phase, adding latency.

Why KV Cache Cannot Be Reused Across Models

Prefix caching works within a single model: identical prefixes reuse previously computed KV cache. Across models, KV cache is tied to model-specific parameters (layer count, attention heads, hidden dimensions), making direct reuse impossible even for identical input text.

MapFlow: Reusing Attention Maps as Transferable Intermediate State

MapFlow shifts focus to attention maps — matrices showing how much attention each token assigns to others. When two models share a tokenizer and process the same context, token positions align, providing a natural basis for reuse. The receiving model still uses its own value representations and subsequent computation, but leverages the sender's attention structure.

Mapping and Partial Recomputation

Because attention maps differ between models, MapFlow combines partial recomputation with lightweight learned mappings:

The receiving model computes its first few layers normally to establish its hidden states.

In later layers suited for reuse, the sender's attention maps are mapped to the receiver: layer mapping aligns different depths, and head mapping learns linear combinations to adapt differing head counts.

Only mapping weights are trained; both base models remain frozen. In the Ministral 8B → 3B configuration, added weights are < 0.001% of the receiver's parameters.

Mappings are calibrated offline and applied to online requests.

System Components for Practical Efficiency

1. Importance-Based Sparse Block Retention

Full attention matrices grow quadratically with sequence length. MapFlow compresses attention maps into block-sparse format by selecting important regions based on cumulative attention probability. Regular 2D blocks enable efficient GPU access. Compression proceeds in stages — computing block-level statistics then extracting retained blocks — avoiding materialization of the full dense matrix.

2. Prefix-Hash Cache Management

Different layers and positions retain varying numbers of sparse blocks. MapFlow indexes these states with a prefix hash that includes preceding context, allowing the receiver to locate relevant cached blocks. Cache retrieval overlaps with early-layer recomputation. A single sender's cached state can serve multiple downstream models.

3. Fused Mapping and Value Aggregation Kernel

Reconstructing the receiver's full attention matrix before multiplying with values would reintroduce large intermediate tensors. MapFlow's fused Triton kernel reads sparse blocks directly, performs the linear combination of attention heads on-chip, and immediately multiplies with the receiver's values, reducing global memory traffic and intermediate storage.

The system integrates with vLLM and supports PagedAttention, chunked prefill, and variable-length sequence packing.

Experimental Evaluation

Experiments ran on NVIDIA A800 80GB GPUs using Qwen3 and Ministral3 model families across four model-pair configurations (e.g., 14B→8B, 8B→3B) on QA, dialogue, and summarization tasks.

Prefill Latency vs. Quality Trade-off

MapFlow reduces prefill latency by 15%–20% compared to full prefill while preserving generation quality on evaluated tasks. Against FlexPrefill, it achieves a further 7%–12% latency reduction at equal or better quality.

Multi-Turn Dialogue Routing Throughput

Using LMSYS and ShareGPT4 dialogue data to simulate routing workloads, under the same P90 first-token latency constraint, MapFlow increases request throughput by up to 25% , demonstrating improved online serving capacity.

Three-Model Workflow Acceleration

In a pipeline where a 14B model plans, an 8B model refines, and a 3B model summarizes — both downstream models sharing the 14B model's attention state — each workflow round saves 0.48–0.73 seconds , yielding up to 5.4% end-to-end speedup .

Conclusion

MapFlow demonstrates that attention maps serve as valuable cross-model intermediate state. Beyond passing text, models can reuse already-computed token dependency relationships. Through mapping, compression, cache management, and fused execution, this reuse translates into measurable latency and throughput gains for multi-model inference.

Let one context computation serve multiple subsequent models — a possibility MapFlow opens for efficient multi-model inference.

Paper: Accelerating Cross-Model LLM Inference via Attention Map Reuse (accepted at SIGMOD 2027). ArXiv: https://arxiv.org/abs/2610.14688 Code:

https://github.com/PKU-DAIR/MapFlow