How Tair‑KVCache‑HiSim Simulates LLM Inference 390 000× Faster with <5% Error

Background

Large‑model inference in the emerging “agent era” can no longer rely on a single‑GPU in‑memory KV‑Cache. Multi‑level KV‑Cache (SSD → Host DRAM → GPU HBM) introduces a high‑dimensional configuration space that spans model architecture, hardware platform, inference engine, and cache policies. Optimizing latency, throughput and cost while satisfying service‑level objectives (SLOs) is a core challenge for large‑scale deployment.

Key Technical Challenges

Complex request lifecycle : Tokenization, scheduling, prefill, decode and post‑processing are tightly coupled with cache state and hardware resources.

Strong component coupling : Scheduler decisions affect cache prefetching; cache hit rates influence compute load; batch composition impacts GPU kernel efficiency, leading to error amplification in naive models.

Non‑linear step latency : Latency depends on model depth, attention heads, quantization, parallelism, GPU type and dynamic request state, making coarse‑grained models inaccurate.

High‑dimensional configuration search : Exhaustive evaluation of parallelism, batch size, cache strategy and quantization is infeasible; efficient Pareto‑front exploration is required.

Simulation Requirements

Reproduce the full request lifecycle from arrival to response.

Model realistic workloads, multi‑node routing and per‑stage modular behavior.

Provide component‑level high‑fidelity latency models that can be independently validated.

Predict fine‑grained per‑step latency for heterogeneous batches.

Support SLO‑constrained configuration space exploration with multi‑objective optimization.

HiSim Architecture

HiSim is a lightweight, high‑accuracy simulator that runs on a general‑purpose CPU without deploying models to GPUs. It consists of three cooperating components:

Workload Generator : Generates synthetic traces or replays timestamped real traces. Supports random datasets and timestamped datasets for multi‑turn and agent scenarios.

Global Router Simulator : Dispatches requests to workers using strategies such as random, round_robin, cache_aware, power_of_two and bucket.

Inference Engine Simulator : Models the internal behavior of a single inference instance, including scheduling, KV‑Cache management and batch execution.

Inference Engine Simulator Sub‑Modules

SchedulerSimulator accurately reproduces the scheduling logic of frameworks like SGLang and vLLM. It maintains four queues—Waiting, Prefetch, Running and Swapped—and applies prefetch policies ( best_effort, wait_complete, timeout) to decide whether a request can be scheduled.

KVCacheManagerSimulator is the first open‑source simulator to model three‑level KV‑Cache (L3 SSD → L2 Host DRAM → L1 GPU HBM). It handles prefix matching, asynchronous prefetch, eviction policies (LRU, LFU, custom) and the bandwidth/capacity differences of each level.

BatchRunnerEstimator provides fine‑grained, generalizable step‑latency prediction. Each request is represented as a tuple (cache_len, input_len). The estimator supports:

Sampling‑based interpolation/regression models built from offline profiling.

Operator‑level latency composition using compute‑ vs. memory‑bound classification.

Roofline‑based theoretical bounds and scale‑factor regression for unseen hardware.

Multiple back‑ends (e.g., aiconfigurator) can be swapped at runtime.

Global Clock & Event‑Driven Model provides a unified virtual global clock that drives all modules, allowing overlapping CPU scheduling, GPU execution and KV‑Cache transfers to be modeled accurately.

Independent Validation Interfaces

BatchRunnerEstimator : Compare predicted step latency against micro‑benchmarks for specific (cache_len, input_len) pairs and compute mean absolute percentage error (MAPE).

SchedulerSimulator : Replay real scheduler logs and verify queue residency times and ordering.

KVCacheManagerSimulator : Replay cache event traces from production runs and validate hit rates, prefetch volumes and eviction triggers.

Performance Evaluation

Speed

On a typical production workload HiSim reduces evaluation cost by a factor of 1/390,106 , shrinking the time needed for performance assessment from days of GPU‑based measurement to minutes on a commodity CPU.

Accuracy

Step‑latency : Across 958 heterogeneous batches the average prediction error is 4.24 % .

End‑to‑end system metrics : Using an A100‑SXM4‑80GB with SGLang v0.5.6 on the ShareGPT multi‑turn dataset, HiSim’s predictions for first‑token latency (TTFT), per‑token latency (TPOT) and throughput stay within <5 % of measured values for four KV‑Cache configurations (IDLE, DEVICE, HOST, DISK).

Future Outlook

KV‑Cache simulation not only optimizes current deployments but also guides the evolution of AI infrastructure. As new model families (e.g., Mamba, hybrid attention), sparsity techniques and speculative decoding emerge, a “hardware‑first” approach becomes untenable. Future systems must co‑design compute, memory hierarchy and interconnects driven by workload characteristics, with high‑fidelity simulation serving as the decision engine. Ongoing work will explore “KV‑Cache‑driven soft‑hardware co‑evolution” in greater depth.