Why Cache Hit Rate Beats Model IQ in AI Programming Costs

Imagine a brilliant but forgetful lawyer who charges by the hour; each consultation requires you to recount the entire case, inflating time and cost. Giving the lawyer a notebook that records 99.7% of the conversation lets you add only new information, cutting the interaction to seconds and maximizing value. This analogy illustrates the core argument: in AI programming, cache hit rate determines both latency and billing more than model "IQ".

Hierarchical caching is the practical implementation of the notebook. In a real‑time inference service handling thousands of sessions, GPU memory cannot store every session’s notes, so the system layers storage across GPU, RAM, and external disks, each with different access speeds. The article cites three recent PRs to the popular SGLang inference framework that fix specific caching bugs: (1) notes sometimes never get archived, (2) archived notes become invisible to the retrieval engine, and (3) new notes can be attached under an unarchived old note, corrupting the cache structure and potentially crashing the process.

The engineers who submitted those PRs are the same company behind Agnes AI, and their work pushes the cache‑hit price to be 1/120 of the miss price , a factor far beyond competitors such as Kimi K3 (10×) and DeepSeek V4 Flash (50×). This price advantage is not marketing hype but a direct consequence of the cost structure.

Model performance metrics

Agnes 2.5 Pro Alpha processes 8.16 trillion tokens per week (5.10 trillion text, 3.06 trillion multimodal). In the Terminal‑Bench v2.1 suite it achieves a 67% pass rate, and its Intelligence Index scores 39, ranking ninth among comparable models.

Task‑level evaluation

Four representative tasks were used to probe the model’s practical abilities:

Portfolio website generation : a 27‑image (528 MB) collection was given without any instruction to create thumbnails. The model automatically produced previews/ (6.7 MB) and thumbs/ (1.0 MB) directories, and generated a photo_info.json with EXIF metadata, demonstrating proactive handling of large assets.

AI cost tracker : the model devised a "Standard Output Unit" (SOU) to normalize disparate pricing schemes (subscription, pay‑per‑use, free quota) and displayed costs per 1 000 SOU, handling edge cases like zero output by showing Infinity instead of NaN.

Pixel‑soccer game : the prompt only specified five penalty rounds. The model added realistic early‑termination logic (when one side’s lead exceeds remaining kicks) that was missing from the prompt, preventing a nonsensical five‑round continuation.

Constraint‑aware coding : given a rule forbidding external network resources, the model wrote pure Canvas code for charts, correctly handling device pixel ratios and avoiding hidden CDN calls, as verified by a full‑text search that returned zero matches for http, cdn, <script src, and <link.

These four details proved more persuasive than the overall benchmark score because they reveal where the model reliably adds value and where its boundaries lie.

Rollback incident and checkpoint handling

During the portfolio task the model performed a rollback using a single backup file ( index-backup-v1.html), losing intermediate versions. The client’s lack of a snapshot layer meant the rollback could only restore that one baseline. The subsequent log shows the model repeatedly attempting to reconstruct missing versions, finally annotating the Python code with a comment confirming the loss.

This behavior highlights the importance of a proper checkpoint system (sometimes called snapshot ) that records incremental states without polluting the user’s Git history, a feature common in mature AI‑coding clients.

Log‑driven evidence of cache impact

Analyzing 22 request logs revealed a direct correlation between cache hit rate and first‑token latency: hits above 95% yielded latencies of 2.9–3.4 seconds, while the two low‑hit cases (11.7% and 13.4%) showed latencies of 13–14.7 seconds. The model processed 26.99 million input tokens but produced only 112 k output tokens (a 241:1 input‑output ratio), confirming that most cost is incurred on the input side and that improving cache hit rate dramatically reduces both time and expense.

In conclusion, the analysis shows that when AI‑coding models become cheap or free, the scarce resource shifts from compute to verification. Users should allocate AI effort to tasks with objective, observable outcomes (tests, numeric calculations, clear visual feedback) and treat ambiguous or “looks‑right” work as a potential hidden cost.