Deep Dive into Agent Harness: Unpacking the Architecture Behind AI Agents

What Is an Agent Harness?

Agent Harness is the complete software stack that surrounds a large language model (LLM) to make autonomous agents possible. It includes orchestration loops, tool integration, memory, context management, state persistence, error handling, and security safeguards. Anthropic’s Claude Code SDK explicitly calls this stack the “agent harness,” and OpenAI’s Codex team treats “agent” and “harness” as synonymous concepts that enable LLMs to be useful.

Vivek Trivedy (LangChain) summarizes the distinction: if you are not the model itself, you are the harness. An "agent" is the emergent behavior—goal‑directed, tool‑using, self‑correcting entity—while the harness is the machinery that produces that behavior.

Beren Millidge (2023) likens a bare LLM to a CPU without memory, storage, or I/O. The context window is fast but limited memory, an external database acts as a hard drive, tool integrations are device drivers, and the harness functions as the operating system.

Three Engineering Layers

Prompt Engineering : designs the instructions the model receives.

Context Engineering : decides what the model sees and when.

Harness Engineering : combines the first two and adds full application infrastructure—tool orchestration, state persistence, error recovery, verification loops, safety guards, and lifecycle management.

Production‑Grade Harness: 12 Core Components

Orchestration Loop : Implements the think‑act‑observe (TAO) or ReAct cycle—assemble prompts, call the LLM, parse output, execute tool calls, feed results back, and repeat.

Tools : Defined by schema (name, description, parameter types) and injected into the LLM context. They handle registration, validation, sandboxed execution, result capture, and formatting for the model.

Memory : Operates on short‑term (single‑session dialogue) and long‑term (persistent across sessions). Anthropic uses CLAUDE.md files; LangGraph stores JSON per namespace; OpenAI supports SQLite or Redis‑backed sessions.

Context Management : Prevents "context decay" where critical information placed in the middle of the window degrades performance by >30 % (Chroma study, corroborated by Stanford’s “Lost in the Middle” paper). Strategies include compression, observation masking, and on‑demand retrieval.

Prompt Construction : Layers system prompts, tool schemas, memory files, dialogue history, and the current user message, placing high‑signal tokens at the start and end of the prompt.

Output Parsing : Modern harnesses rely on structured tool_calls objects rather than free‑form text. If a tool call is present, the loop executes it; otherwise the output is final.

State Management : LangGraph models state as a typed dictionary flowing through graph nodes, with reducers merging updates. Checkpoints enable interruption recovery and time‑travel debugging. OpenAI offers four mutually exclusive strategies (in‑process memory, SDK session, server‑side Conversations API, lightweight previous‑response linking).

Error Handling : A 99 % per‑step success rate over a 10‑step workflow yields only ~90 % end‑to‑end success, so errors accumulate quickly. Harnesses categorize errors into transient (with back‑off), LLM‑recoverable (returned as tool messages), user‑fixable, and unexpected (bubbled up for debugging). Stripe’s production harness caps retries at two.

Safety & Guardrails : OpenAI SDK implements three layers—input guardrails (first agent), output guardrails (final output), and tool guardrails (each call). Anthropic separates permission execution from model reasoning, gating ~40 tools through staged checks and explicit user confirmation for high‑risk actions.

Verification Loop : The dividing line between toy demos and production agents. Anthropic recommends rule‑based feedback, visual feedback (Playwright screenshots), and LLM‑as‑judge (a separate sub‑agent). Claude Code’s creator Boris Cherny notes that giving the model a way to verify its own work can improve quality 2‑3×.

Sub‑Agent Orchestration : Supports forked execution (byte‑level copy), teammate panels (file‑based mailbox), or worktree models (independent git branches). OpenAI treats agents as tools (expert sub‑tasks) or hand‑offs (expert takes full control). LangGraph nests sub‑agents as embedded state graphs.

Step‑by‑Step Loop Walkthrough

Prompt Assembly : Harness builds the full input—system prompt, tool schemas, memory files, dialogue history, and the current user message. Critical context is placed at the prompt’s boundaries (per “Lost in the Middle” findings).

LLM Inference : The assembled prompt is sent to the model API, which returns text, tool‑call requests, or both.

Output Classification : If only text is returned, the loop ends. If a tool call is present, execution proceeds. If a hand‑off is requested, the current agent is swapped and the loop restarts.

Tool Execution : Harness validates parameters, checks permissions, runs the tool in a sandbox, and captures results. Read‑only calls may run concurrently; write operations are serialized.

Result Packaging : Tool results are formatted as LLM‑readable messages. Errors are captured and returned as error results for the model to self‑correct.

Context Update : Results are appended to the dialogue history. When the context window nears its limit, the harness triggers compression.

Loop Continuation : Return to step 1 until a termination condition is met (no tool call, max rounds, token budget exhausted, guardrail trip, user interrupt, or safety refusal).

Seven Design Decisions Every Harness Faces

Single‑agent vs. Multi‑agent : Start with a single agent; split only when tool overlap exceeds ~10 or distinct task domains emerge.

ReAct vs. Plan‑Execute : ReAct interleaves reasoning and action (flexible but costly). Plan‑Execute separates planning from execution; LLMCompiler reports a 3.6× speedup over sequential ReAct.

Context Window Management : Five production strategies—time‑based eviction, dialogue summarization, observation masking, structured notes, and sub‑agent delegation. ACON research shows that preserving reasoning traces while discarding raw tool output reduces token usage by 26‑54 % while keeping >95 % accuracy.

Verification Loop Design : Deterministic testing (unit tests, code checkers) provides a ground‑truth baseline; LLM‑as‑judge captures semantic issues but adds latency. Martin Fowler’s framework separates “guides” (pre‑action hints) from “sensors” (post‑action feedback).

Permission & Security Model : Choose between permissive (fast, higher risk, auto‑approve) and restrictive (slow, lower risk, manual approval) based on deployment context.

Tool Scope Strategy : Fewer tools generally improve performance. Vercel removed 80 % of its tools and saw better results; Claude Code uses lazy loading to cut context by 95 %.

Harness Thickness : Decide how much logic lives in the harness versus the model. Anthropic favors a thin harness that shrinks as models improve, while graph‑based frameworks keep more explicit control.

Why Harness Matters

Changing only the harness can shift an agent’s ranking on TerminalBench 2.0 by more than 20 positions, demonstrating that the infrastructure, not the model weights, drives performance differences. As models evolve, harness complexity should decrease, but the need for context management, tool execution, state persistence, and verification remains.

When a production agent fails, the fault is rarely the model itself; it is often the surrounding harness.