Agent Harness Explained: A Deep Dive into Agent Architecture

What Is an Agent Harness?

First coined in early 2026, the term “Agent Harness” refers to the complete software stack that surrounds a large language model (LLM). It includes the orchestration loop, tool integration, memory, context management, state persistence, error handling and security guardrails. Anthropic’s Claude Code SDK explicitly calls its SDK the "agent harness" and OpenAI’s Codex team treats the agent and harness as synonymous concepts.

Key Distinction

According to Vivek Trivedy (LangChain) an agent is the emergent, goal‑directed behavior that interacts with users, while the harness is the machinery that makes that behavior possible. In practice, developers often say they built an agent when they actually built a harness and then attached a model.

Analogy

Beren Millidge (2023) likens a bare LLM to a CPU without memory, storage or I/O. The context window is the RAM, an external database is the hard drive, tool plugins are device drivers, and the harness functions as the operating system that coordinates everything.

Three Engineering Layers

Prompt Engineering : designs the instructions the model receives.

Context Engineering : decides what the model sees and when.

Harness Engineering : combines the first two and adds tool orchestration, state persistence, error recovery, verification loops and lifecycle management.

Production‑Grade Harness: Twelve Core Components

Orchestration Loop – implements the TAO (Think‑Act‑Observe) or ReAct cycle.

Tools – schema‑defined utilities injected into the LLM context.

Memory – short‑term dialogue history and long‑term persistent stores (e.g., Claude Code’s CLAUDE.md, LangGraph JSON Store, OpenAI SQLite/Redis sessions).

Context Management – strategies to avoid “context decay” (Chroma study, Stanford “Lost in the Middle” paper).

Prompt Construction – layered system prompt, tool schema, memory files, dialogue history, and user message.

Output Parsing – modern harnesses expect structured tool_calls objects; fallback parsers like RetryWithErrorOutputParser remain for edge cases.

State Management – LangGraph’s reducer‑based state graph, OpenAI’s four mutually exclusive session strategies, Claude Code’s git‑commit checkpoints.

Error Handling – per‑step success probability composes multiplicatively (e.g., a 99 % step success over ten steps yields ~90 % end‑to‑end success). Harnesses classify errors as transient, LLM‑recoverable, user‑fixable or unexpected.

Guardrails & Security – input, output and tool guardrails; Anthropic separates permission checks from model reasoning; a circuit‑breaker can abort the agent instantly.

Verification Loop – rule‑based tests, visual checks (Playwright screenshots), or LLM‑as‑judge sub‑agents (Claude Code’s claim of 2‑3× quality boost).

Sub‑Agent Orchestration – Fork, Teammate, Worktree models (Claude Code) and agent‑as‑tool or hand‑off patterns (OpenAI SDK).

Framework Implementations – Anthropic’s Claude Agent SDK (single query() exposing the harness), OpenAI’s Agents SDK (Runner class with async/sync/stream modes), LangGraph (explicit state‑graph), CrewAI (role‑based multi‑agent crew), AutoGen (dialogue‑driven orchestration).

Step‑by‑Step Loop Walk‑through

Prompt Assembly : system prompt + tool schemas + memory files + dialogue history + user message. Critical context is placed at the beginning and end to avoid “lost in the middle”.

LLM Inference : the assembled prompt is sent to the model API, which returns text, tool calls, or both.

Output Classification : if only text is returned, the loop ends; if tool calls are present, execution proceeds; if a hand‑off is signaled, the current agent is swapped.

Tool Execution : each call is validated, permission‑checked, sandboxed, and its result captured. Read‑only calls may run concurrently; write calls are serialized.

Result Packaging : tool results are formatted for the LLM; errors are returned as structured error messages for self‑correction.

Context Update : results are appended to the dialogue history; when the context window nears its limit, the harness triggers compression (e.g., Claude Code’s summarization that retains architectural decisions and unresolved bugs).

Loop Continuation : return to step 1 until a termination condition is met (no tool call, max rounds, token budget exhausted, guardrail trip, user interrupt, or safety refusal).

Design Decisions and Trade‑offs

Single‑agent vs. multi‑agent: start with a single agent; split only when tool overlap exceeds ~10 or distinct task domains emerge.

ReAct vs. plan‑execute: ReAct interleaves reasoning and action (flexible but higher per‑step cost); plan‑execute separates planning, yielding up to 3.6× speedups (LLMCompiler benchmark).

Context‑window strategies: time‑based eviction, dialogue summarization, observation masking, structured notes, sub‑agent delegation. ACON research shows preserving reasoning traces while discarding raw tool output cuts token usage by 26‑54 % with >95 % accuracy.

Validation approaches: deterministic rule‑based tests vs. LLM‑as‑judge; the former offers hard truth, the latter captures semantic issues at the cost of latency (Martin Fowler’s ThoughtWorks framework).

Security posture: permissive (fast, higher risk) vs. restrictive (slow, safer); choice depends on deployment scenario.

Tool‑scope policy: fewer tools improve performance; Vercel removed 80 % of its tools and saw better results; lazy‑loading can reduce context by up to 95 % (Claude Code).

Harness thickness: balance how much logic resides in the harness versus the model. Anthropic bets on a thin harness that lets model improvements absorb functionality, while graph‑based frameworks keep explicit control.

Empirical Evidence

LangChain demonstrated that merely swapping the underlying harness (without changing model weights) moved a system from outside the top 30 to rank 5 on TerminalBench 2.0. Another research project let the LLM itself optimise the infrastructure, achieving a 76.4 % success rate—surpassing manually engineered systems.

Conclusion

Even as LLM capabilities grow, the harness remains indispensable: it manages scarce context, orchestrates tool calls, persists state, and validates work before failures cascade. When an agent misbehaves, the fault often lies in the harness design rather than the model itself.