Jev Engineering: Rethinking Coding Agent Architecture Beyond KV Cache

Introduction

The article translates a technical memo based on design notes from Diogo Almeida, founder of TypeSafe, titled "Jev Engineering for Coding Agents." It opens with a thought experiment: if language models had no KV cache, how would you redesign coding agents? This question exposes how the economics of KV caching have silently shaped today's agent architectures into a homogeneous pattern: a while loop around an LLM, a giant prompt stuffed with all tool schemas, and an ever-growing append-only transcript.

Six Symptoms of KV Cache Tyranny

The memo identifies six systemic defects inherited from the append-only transcript model:

1. Routing failure: Handing control back to a frontier model forces re-processing of the entire context, making mixed routing more expensive than pure frontier use.

2. Tool context crowding: All tool schemas must reside permanently in the system prompt, wasting tokens and degrading selection accuracy.

3. Rigid compression: Static summarization assumes all future turns need the same state, discarding details later required for debugging.

4. Sub-agent underuse: Deciding what context to pass to a child agent and how to merge results back is prohibitively difficult.

5. Full session restarts: When transcripts hallucinate or drift, users restart, discarding valuable history along with noise.

6. Built-in feature dilemma: Every built-in capability permanently consumes context window, forcing a false choice between batteries-included and minimal tools.

Token Consumption Reality

Empirical breakdown (input-weighted view) from a typical CLI coding agent session reveals:

File reading: 30–40% (largest pool; files re-read every turn)

Codebase retrieval: 10–18% (grep, glob, directory listings with noisy output)

Command execution output: 10–20% (stack traces and logs balloon context)

System prompt, tool schemas, AGENTS.md: 5–12% (fixed per-turn overhead)

Session history replay: amplifier (causes all above to be re-billed each turn)

Reasoning & planning: 5–15% (higher for deep debugging)

Code writing & editing: 4–10% (Git diff and str_replace formats are concise)

User explanations: 2–5%

Microsoft's fastcontext study on GPT-5.4 trajectories corroborates: file reads and searches accounted for 56.2% of tool calls and 46.5% of main-agent tokens. The conclusion: the biggest efficiency lever is smarter code retrieval and recall , not stronger models or better diff formats.

Permission Engine & Tool Router

Two immediately adoptable improvements:

Programmable permission engine: Replace opaque classifiers with rule-based queries that can statically analyze script contents before execution (e.g., deny network egress unless task scope is "deploy").

Harness as tool router: The main model describes intent in natural language; Harness uses typed Jev judgments to select the best tool (top-k) and assemble its parameters, eliminating the need for the model to hold hundreds of schemas in context.

Meta-Attention: Context as a Decision

Instead of passive accumulation, every user prompt triggers two Jev decisions:

Cache reuse vs. fresh assembly: Explicit cost-aware choice between reusing the KV prefix or building a new context from scratch.

Visibility Ladder per chunk: For each atomic context chunk (tool input, output, reasoning step, QA pair), Jev assigns one of four levels: hide, short, long, full. This implements query-aware compression : a 2400-line grep log can be distilled to 12 key lines for a specific debugging query, yet hidden entirely for an unrelated follow-up, without ever being physically deleted from global state.

The memo envisions heatmap-driven dynamic subsampling of retrieval output at any precision the budget allows.

Revitalizing Model Routing & Sub-Agents

Dynamic context assembly unlocks viable routing: when Harness can craft a small, purpose-built context slice for a sub-task, a cheap model no longer needs to swallow the full history, and its results can be merged back as scored chunks without forcing the frontier model to re-read all low-level generations. The routing math (public pricing: Opus $5/$25 per M in/out, Sonnet $3/$15) shows that mixed routing (Opus → Sonnet → Opus) costs 3X + 20Y + 8Z vs. pure Opus 25Y + 5Z. With realistic distributions (X=0.65, Y=0.12, Z=0.23), pure Opus costs 4.15 units vs. 6.19 for mixed routing — routing increases cost unless downstream context reload and upstream re-read overhead are eliminated.

Sub-agent spawn cost drops to near-zero when context slicing is automated, enabling massive parallelism. Harness manages concurrency with a shared state store featuring read/write locks; pure read-only tasks never contend.

Tool & Skill Redesign: Progressive Disclosure

The memo proposes a three-layer buffer between static full-schema loading and fully on-demand skills:

Cheap panorama: Model sees a one-line summary of every capability.

Schema on demand: Only when intent matches does Harness fetch the full schema for the selected tool(s).

Zero permanent pollution: After task completion, all tool details are unloaded from the main context.

This dissolves the batteries-included debate: idle cost approaches zero, so hundreds of tools and thousands of documentation pages can ship by default.

Conditional Instructions (AGENTS.md)

Instead of loading the entire AGENTS.md every turn, rules are bound to concrete triggers: load frontend style guide only when editing UI code; load a directory's "footguns" file only when the terminal enters that subtree. These conditional chunks are compression-immune — they are re-anchored every assembly round as long as their trigger condition holds, unlike skills which eventually get summarized away.

Security-Aware Routing

Routing decisions add a third dimension: data trust . Each sub-task's potential file accesses are classified, and policy drives model selection:

Public docs / open-source deps → Open tier: any model, cheapest first.

Core business logic → Standard tier: vetted first-party providers.

Secrets, infra config → Restricted tier: only compliant top-tier frontier models.

Proprietary research code → Custom tier: exclude specific competitors.

Policy-driven routing encodes compliance into configuration, not engineer memory.

Read-Only Background Pipelines

Emerging workflows (live HTML progress boards, cross-vendor code review, continuous eval generation, ELI5 explainers, mobile progress dashboards) share a physical trait: they are pure read-only functions of the current codebase state. Expensive dependency/impact analysis is performed once and shared across all background consumers, dropping their cost by orders of magnitude. Explicit read/write typing in the state store gives agents "near-superpower" concurrency.

Candidate Native Integrations

The memo lists high-potential open-source tools with Jev-native integration sketches:

headroom (context compressor) → attach classifier to verify retained facts post-compression.

rtk (tool output compressor) → ship official prompt teaching base model its custom format.

ast-grep (structural search) → load rule handbook once, batch-generate N AST queries, filter by relevance.

ast-outline (syntax outline) → hierarchical drill-down on demand.

fastcontext (exploration sub-agent) → route retrieval to it or use its structured index instead of regex search.

fff (fast file/content search) → resident memory index, frequency-weighted, outperforms ripgrep in long sessions.

Conclusion

The core loop of a coding agent is trivial; the lever is what the framework feeds the model each turn . Today that decision is hijacked by KV-cache economics. Removing KV cache in the thought experiment reveals a new architecture: fully explicit, typed state; a dedicated decision model (Jev) that dynamically assembles context via a visibility ladder; routing that accounts for context-rebuild overhead; progressive tool disclosure; conditional, compression-immune instructions; security-driven model selection; and shared read-only pipelines. This does not require a stronger base model — it requires treating the context window as a precision workshop assembled per intent, not an accidental landfill of history.