DeepSeek Harness: How 99% Cache Hits Validate Request Prefix Stability

Understanding the 1/30 Price Gap and 99% Hit Rate

DeepSeek V4 Flash peak pricing shows a 30x difference between cache hit (¥0.10 per million tokens) and miss (¥3.00). At 99% hit rate, input cost drops to ¥0.129 per million tokens (99% × 0.10 + 1% × 3.00), roughly 1/23.3 of the all-miss cost. With 100k output tokens adding ¥0.90, total task cost becomes ~¥1.03 vs ¥3.90 all-miss — a 3.8x difference. However, the 99% figure comes from specific public tests (e.g., Pi + DeepSeek V4 Flash) and DeepSeek's own documentation notes cache is best-effort, not guaranteed. Hit rate depends on request structure, task shape, and provider cache behavior.

Cache Matches Prefixes, Not "Close Enough"

An agent request splits into routing (provider/model/reasoning) and model input (stable system prompt + ordered tool schemas → existing history → new messages). Changing model or provider switches cache spaces. Unlike Redis where keys are explicit, provider cache matches token prefixes exactly: one token change breaks the common prefix. Details that break prefixes include: tools registered in non-deterministic order, timestamps/random IDs injected into system prompt, schema field order drift, history compression rewriting old messages, and permission/sandbox/MCP tool changes invisible in logs. OpenAI's Codex agent loop learned this: early unstable MCP tool ordering caused cache drops; they fixed it by appending config updates as new messages instead of rewriting prefixes.

DSH as a "Request Compiler"

Examining DSH snapshot 0a53fb55bea1 (@deepseek-ai/dsh-agent-loop 0.1.2-alpha.2), the framework acts like a compiler: plugins supply prompt fragments, tools, and runtime context; system prompt assembles them into deterministic input; agent loop generates message history from session and emits request config to provider. This determinism underpins the 99% hit rate.

Prompts and Tools: Fix Order First

DSH's system-prompt module orders prompt fragments by explicit order, then by code-unit name (locale-independent). Tools sort by name by default; toolOrder config places unlisted tools in <unlisted-tools> slot then sorts by name. Invalid configs (missing tools, duplicates, missing fallback) cause assembly errors. Variables referencing unregistered or empty values halt assembly. These strict checks prevent silent prefix drift.

Session: Record Facts First, Derive Chat History

DSH's Session is an append-only SessionEvent log (user messages, model streaming chunks, full replies, tool calls, tool results). deriveMessages() projects the next model-visible history from these facts. A hard constraint: "Model-visible means already recorded." Everything in the model request must be reconstructible from the log. Agent loop includes request-rebuild tests guarding this boundary. Unlike Pi's long-task Context (working set), DSH's Session retains original events even after compression — compressed content replaces model-visible prefix, but raw events stay for audit. Normal tool round-trips append to old history: original prefix → assistant/tool-call → tool/result → next model request , preserving prior tokens.

Runtime Context: Append Only on Change

Runtime context (sandbox policy, approval policy, sub-agent delegation) enters model history via RuntimeContextProjection. System prompt renders a full snapshot; projection compares with previous effective snapshot from session: (1) no prior context and current empty → no message; (2) snapshot identical to previous → no message; (3) content changed → append sourced user/message; (4) context cleared → append explicit clear marker. This avoids re-inserting unchanged state each turn and avoids rewriting old messages. Boundary: cross-session reuse still limited because first user message position shifts the common prefix.

Request Header: Track What Changed

Each request emits a normalized request/header recording provider, model, reasoning config, adapter defaults, full system prompt, and ordered tool schemas (compared item-by-item). Header types: initial (first request), resume (restored from persisted session), change (fixed config changed), series (message projection started new sequence, config unchanged). If both occur, change carries startsSeries: true. This header isn't the provider's cache key but lets operators pinpoint why hit rate dropped: did model/reasoning change? Which system prompt segment? Tool added/removed/reordered? Compression started new series?

Source Code Tests Hit Detection, Not 99%

DSH's request-cache.e2e.ts (requires real DeepSeek API key) inserts a long system prompt, runs a lookup tool, then a follow-up — generating at least three model requests. It reads usage on each assistant message; from second request onward it asserts cacheReadTokens > 0 (mapped from provider's prompt_cache_hit_tokens). This proves multi-step requests generated from log and appended continuously are recognized by the real provider. The test does not assert 99% and cannot guarantee hits for all profiles, tool combos, or cold starts. Unit tests verify message array appends; end-to-end test confirms provider actually hits cache — connecting "prefix should be stable" from code structure to real receipts.

Cross-Session Reuse Gap

DSH currently doesn't automatically place all stable content at the very front. In preStep(), agent loop claims current messages first, then appends runtime context. Two new sessions in same workspace share project rules, sandbox, approvals — but different first user messages push stable content behind the first user token, preventing cross-session common prefix. Discussion #4749 reports: new sessions with different first messages ~42% hit; same first message ~99.8%; after local plugin reorder, new sessions ~94%, TTFT from ~10s to ~3s. These are community experiments, not official benchmarks. Moving non-user content before first user message isn't automatically correct: time, terminal state, temporary permissions, retrieval results may come from plugins and aren't necessarily stable. Reordering for cache may alter instruction recency, dialogue semantics, safety policy position. The gap: runtime lacks a formal context stability interface. Needed layers: (1) Stable baseline — same profile/workspace/config digest, enters common prefix; (2) User input — keep FIFO, don't swap for hit rate; (3) Dynamic context — terminal state, temp retrieval, per-step auth, append with current step; (4) Updated facts — policy/permission/compression baseline changes, explicitly append new version and invalidate old.

When Hit Rate Drops, Check What Changed in Request

If production agent drops from ~95% to ~40% hit rate, it's not just billing — tool list may have drifted, prompt injection order changed, compressor rewrote earlier history, plugin injected random values into stable zone. Debugging steps: (1) Separate cold vs hot requests; group by provider, model, profile, task type — don't average different requests; (2) Cross-reference request/header and session projection to see if new request generation started; (3) Locate first diff in system, tools, or messages — note content vs order vs serialization change; (4) Combine cache read/miss tokens, output tokens, TTFT, retries to compute full task cost. Production can emit a prefixDigest (not provider cache key) identifying "runtime believes these requests share same prefix" — include model routing, system prompt, tool names + schema order, request generation. Log fields: provider/model/profile, requestGeneration/prefixDigest, cacheReadTokens/cacheMissTokens/outputTokens, timeToFirstToken/retries, firstDiffArea (system/tools/messages). If digest changed, check if normal release or plugin leaked randomness into stable zone. If digest stable but provider hit rate fell, investigate cold start, cache persistence, server-side eviction, metric mapping. Runtime assembling same request ≠ provider hitting cache — two distinct questions. Some invalidations are correct: tool permissions tightened, model swapped, system prompt security fix, compression new baseline — these should start new generation. Keeping old policy for pretty charts is counterproductive. Cache hit rate can verify runtime stability, not dictate what content must stay unchanged.

Request Facts Belong in Harness

Prompt cache is provider-implemented; request facts and history semantics should stay in Harness. DSH's Session and request/header outlast "99%". If provider cache expires, one round is costlier/slower; but if full request, replacement relations, and model-visible history exist only in provider black box, switching providers, restoring sessions, and incident audits become harder. Clean boundary: Harness stores event facts, request config snapshots, rebuildable history; Adapter aligns provider I/O and usage semantics; Provider handles inference, cache implementation, cache receipts. This captures model-side cache without surrendering system interpretability.

Cache Receipt Reflects Entire Runtime

Back to that first tool call: next request reuses prefix iff tokens unchanged. Why tokens change is decided by the full runtime chain: prompt/tool ordering, session-projected messages, runtime context append strategy, request generation records. DSH didn't invent DeepSeek's KV cache nor promise 99%. Its contribution: making "what exactly was sent to the model this time" something that can be assembled, recorded, rebuilt, and compared. The most useful place for cache hit rate is verifying: given the same facts, can the runtime still assemble the same request? Prices, models, cache rules will change. If runtime can explain where each request came from and which generation changed, that fundamental capability persists across providers.