What Is an Agent Harness? A Deep Dive into AI Agent Architecture

Why a Harness Matters

Building a chatbot that works in a production setting quickly runs into problems: the model forgets actions taken three steps ago, tool calls fail silently, and the context window fills with irrelevant data. The author argues that the failure is not the model itself but the surrounding infrastructure.

Defining Agent Harness

Anthropic’s Claude Code documentation calls the SDK the "agent harness," and OpenAI’s Codex team treats "agent" and "harness" as interchangeable concepts that make an LLM useful. Vivek Trivedy of LangChain summarizes this with a crisp definition: If you are not the model itself, you are the harness.

Three Engineering Layers

The author separates the work into three concentric layers around the model:

Prompt engineering : designing the instructions the model receives.

Context engineering : deciding what the model sees and when.

Harness engineering : the full application stack that adds tools, persistence, error recovery, validation loops, safety guards, and lifecycle management.

Production‑Grade Harness: Twelve Components

A production‑grade harness consists of the following independent components:

Orchestration loop : the TAO/ReAct heartbeat that assembles prompts, calls the LLM, parses output, executes tools, feeds results back, and repeats.

Tools : schema‑defined utilities (name, description, parameters) that are injected into the LLM context; the harness handles registration, validation, sandboxed execution, and result formatting.

Memory : short‑term dialogue history and long‑term persistent stores (e.g., Claude Code’s CLAUDE.md files, LangGraph’s JSON Store, OpenAI’s SQLite/Redis sessions).

Context management : strategies such as compression, observation masking (e.g., JetBrains Junie), instant retrieval, and sub‑agent delegation to keep the token window effective.

Prompt construction : layering system prompts, tool schemas, memory files, conversation history, and the current user message.

Output parsing : modern harnesses rely on structured tool_calls objects; fallback parsers (e.g., RetryWithErrorOutputParser) handle free‑text failures.

State management : LangGraph models state as a typed dictionary flowing through graph nodes with reducers and checkpoints; OpenAI offers session‑based, conversation‑based, and previous‑response‑ID strategies.

Error handling : four error categories—instant retry, LLM‑recoverable, user‑fixable, unexpected—are treated with exponential back‑off or circuit‑breaker logic (Stripe limits retries to two).

Guardrails & security : input, output, and tool guards; Anthropic separates permission execution from model reasoning, using a three‑stage gate for ~40 tools.

Validation loop : rule‑based feedback, visual checks (Playwright screenshots), or LLM‑as‑evaluator sub‑agents; Claude Code’s creator Boris Cherny reports a 2‑3× quality boost when the model can self‑validate.

Sub‑agent orchestration : execution models such as Fork, Teammate, Worktree (Claude Code) or agent‑as‑tool and hand‑off (OpenAI); LangGraph nests sub‑agents as embedded state graphs.

Step‑by‑Step Loop Walk‑through

The author demonstrates a full cycle in seven steps:

Prompt assembly : system prompt + tool schemas + memory files + dialogue history + user message; critical context is placed at the beginning and end (as discovered in the "lost‑in‑the‑middle" study).

LLM inference : the assembled prompt is sent to the model API, which returns text, tool calls, or both.

Output classification : if only text is returned, the loop ends; if tool calls are present, execution proceeds; if a hand‑off is requested, the current agent is swapped and the loop restarts.

Tool execution : the harness validates parameters, checks permissions, runs the tool in a sandbox, and captures the result (read‑only calls may run concurrently, writes are serialized).

Result packaging : tool results are formatted as LLM‑readable messages; errors are returned as structured error objects for self‑correction.

Context update : results are appended to the conversation; when the window limit is approached, compression is triggered.

Loop continuation : return to step 1 until a termination condition is met (no tool call, max rounds, token budget, guardrail trip, user interrupt, or safety refusal).

Termination can happen after a single round for simple queries or after dozens of tool calls for complex refactoring tasks.

Framework Implementations

Major open‑source and vendor frameworks embody the harness pattern:

Anthropic Claude Agent SDK : exposes a single query() function that runs a "dumb loop"; Claude Code follows a collect‑act‑validate cycle.

OpenAI Agents SDK : provides a Runner class supporting async, sync, and streaming modes; the Codex harness adds a three‑tier stack (core, API server, UI) that shares a single harness across interfaces.

LangGraph : models the harness as an explicit state graph with conditional edges (tool call vs end); it supersedes LangChain’s AgentExecutor, which was deprecated for lack of extensibility.

CrewAI : builds role‑based multi‑agent crews (Agent, Task, Crew) with a flows layer that adds deterministic routing and validation.

AutoGen (Microsoft Agent Framework) : dialog‑driven orchestration supporting sequential, concurrent (fan‑out/fan‑in), group chat, hand‑off, and "magentic" accounting patterns.

Design Decision Matrix

The author outlines seven recurring decisions for any harness:

Single‑agent vs multi‑agent (prefer single until tool overlap > 10 or distinct domains appear).

ReAct vs plan‑execute (ReAct offers flexibility; plan‑execute can be 3.6× faster per LLMCompiler).

Context window strategy (time‑based eviction, summarization, observation masking, structured notes, sub‑agent delegation; ACON study shows 26‑54 % token savings with <90 % accuracy).

Validation loop design (deterministic rule‑based tests vs LLM evaluator; Martin Fowler’s guide‑sensor model).

Security posture (lenient fast‑track vs restrictive per‑operation approval).

Tool scope (expose only the minimal set needed for the current step; Vercel’s removal of 80 % of tools improved performance).

Harness thickness (how much logic lives in the harness vs the model; Anthropic moves toward thinner harnesses as models improve).

Empirical Evidence

Several benchmarks illustrate the impact of harness design:

LangChain’s benchmark on TerminalBench 2.0 shows that swapping only the harness moves an agent from outside the top 30 to rank 5.

A research project letting the LLM optimise its own infrastructure achieved a 76.4 % success rate, surpassing manually engineered systems.

With a per‑step success rate of 99 % over a 10‑step flow, the end‑to‑end success drops to ~90.4 %, highlighting error accumulation.

Claude Code’s lazy‑load mechanism reduces context usage by 95 % while preserving functionality.

Conclusion

Agent Harness is not a solved problem or a commodity layer; it is the hard‑core engineering that manages scarce context, captures failures before they cascade, provides persistent memory, and validates work. As models become more capable, harnesses can become thinner, but they will not disappear—every powerful model still needs a harness to orchestrate tools, persist state, and ensure safety. When an agent misbehaves, the first place to look is the harness, not the model.