Deep Dive into Agent Harness: Dissecting the Architecture Behind AI Agents

What Is an Agent Harness?

Agent Harness is a formally named infrastructure layer that wraps a large language model (LLM) with all the non‑model services required for autonomous behavior: orchestration loops, tool integration, memory, context management, state persistence, error handling, safety guardrails, and lifecycle management. Anthropic’s Claude Code documentation calls the SDK the “agent harness” that drives Claude Code, and OpenAI’s Codex team treats “agent” and “harness” as synonymous concepts.

Vivek Trivedy of LangChain succinctly states, “If you are not the model itself, you are the harness.” The distinction is that the agent is the emergent, goal‑directed entity that interacts with users, while the harness is the machinery that makes that behavior possible.

Engineering Layers

Three concentric layers surround the model:

Prompt Engineering : designs the instructions the model receives.

Context Engineering : decides what the model sees and when.

Harness Engineering : combines the first two layers with full application infrastructure (tool orchestration, state persistence, error recovery, verification loops, security, and lifecycle management).

The harness is not merely a wrapper for prompts; it is the system that enables genuine autonomous agent behavior.

12 Production‑Grade Components

Drawing from Anthropic, OpenAI, LangChain, and broader community practice, a production‑ready harness consists of the following components:

Orchestration Loop : implements the Think‑Act‑Observe (TAO) or ReAct cycle—assemble prompts, call the LLM, parse output, execute tools, feed results back, and repeat.

Tools : defined by schema (name, description, parameter types) and injected into the LLM context. Claude Code offers six tool categories (file ops, search, execution, web access, code intelligence, sub‑agent generation); OpenAI’s Agents SDK supports function tools, hosted tools (WebSearch, CodeInterpreter, FileSearch), and MCP server tools.

Memory : short‑term (dialogue history) and long‑term (persistent stores). Anthropic uses CLAUDE.md and auto‑generated MEMORY.md; LangGraph stores JSON per namespace; OpenAI can back memory with SQLite or Redis sessions.

Context Management : mitigates “context decay” where critical information placed in the middle of the window degrades performance by >30 % (Chroma study, corroborated by Stanford’s “Lost in the Middle” paper). Strategies include compression, observation masking, on‑demand retrieval, and sub‑agent delegation.

Prompt Construction : layers system prompts, tool schemas, memory files, conversation history, and the current user message, placing essential context at the beginning and end of the prompt (as discovered in the “Lost in the Middle” analysis).

Output Parsing : modern harnesses rely on the LLM’s native tool_calls object rather than free‑form text, allowing straightforward detection of tool calls, execution, and final answers.

State Management : LangGraph models state as a typed dictionary flowing through graph nodes, with reducers merging updates and checkpoints enabling interruption recovery and time‑travel debugging. OpenAI offers four mutually exclusive strategies (in‑app memory, SDK session, server‑side Conversations API, lightweight previous_response_id linking). Claude Code uses Git commits as checkpoints and progress files as structured drafts.

Error Handling : a 10‑step process with a 99 % per‑step success rate yields an overall success rate of ~90.4 %; errors accumulate quickly. Harnesses classify errors into instant retries, LLM‑recoverable errors, user‑fixable errors, and unexpected errors. Anthropic returns error results from tool handlers to keep the loop alive; Stripe caps retries at two.

Safety Guardrails : three tiers—input guardrails (first agent), output guardrails (final answer), and tool guardrails (per call). A “circuit‑breaker” aborts the agent when triggered. Anthropic separates permission execution from model reasoning, gating ~40 discrete tool capabilities across three phases (trust establishment, per‑call permission check, high‑risk confirmation).

Verification Loop : distinguishes toy demos from production agents. Anthropic recommends rule‑based feedback, visual feedback (Playwright screenshots), and LLM‑as‑judge sub‑agents. Boris Cherny notes that giving the model a way to verify its own work can improve quality 2‑3×.

Sub‑Agent Orchestration : Claude Code supports three execution models—Fork (byte‑level copy of parent context), Teammate (independent terminal panel with file‑based mailbox), and Worktree (isolated Git worktree per agent). OpenAI’s SDK treats agents as tools (expert sub‑tasks) or hand‑offs (expert takeover). LangGraph implements sub‑agents as nested state graphs.

Decision Framework : seven recurring decisions guide harness design, including single‑ vs multi‑agent architecture, ReAct vs plan‑execute loops (LLMCompiler reports plan‑execute is 3.6× faster), context‑window management strategies (time‑based eviction, summarization, observation masking, structured notes, sub‑agent delegation—ACON research shows prioritizing reasoning traces reduces token usage by 26‑54 % while retaining >95 % accuracy), verification design (deterministic test‑based vs probabilistic LLM‑as‑judge), permission model (lenient vs restrictive), tool‑scope policy (fewer tools improve performance; Vercel removed 80 % of tools and saw gains), and harness thickness (balance of logic in harness vs model).

Step‑by‑Step Execution Loop

The article walks through a full loop:

Prompt Assembly : combine system prompt, tool schemas, memory files, conversation history, and user message; place critical context at prompt boundaries.

LLM Inference : send the assembled prompt to the model API; receive text, tool calls, or both.

Output Classification : if only text → terminate; if tool call → proceed to execution; if hand‑off → switch current agent and restart.

Tool Execution : validate parameters, check permissions, run in a sandbox, capture results; read‑only calls may run concurrently, writes are serialized.

Result Packaging : format tool results as LLM‑readable messages; capture errors for self‑correction.

Context Update : append results to conversation history; trigger compression when near window limits.

Loop Continuation : return to step 1 until a termination condition is met (no tool call, max rounds, token budget exhausted, guardrail trip, user interrupt, or safety refusal).

For long‑running tasks spanning multiple windows, Anthropic’s “Ralph Loop” uses a two‑stage approach: an initialization agent sets up environment, progress files, and a Git commit; subsequent sessions read the Git log and progress file to locate the current state, prioritize unfinished high‑value functions, complete them, and write a summarized commit.

Framework Implementations

Different frameworks expose the harness in distinct ways:

Anthropic Claude Agent SDK : a single query() function returns an async iterator of streamed messages; the runtime is a “dumb loop” with intelligence residing in the model. Claude Code follows a collect‑act‑verify cycle.

OpenAI Agents SDK : a Runner class supports async, sync, and streaming modes; the workflow is expressed in native Python (code‑first) rather than a DSL. Codex Harness adds three layers: Core (agent code + runtime), App Server (bidirectional JSON‑RPC), and UI (CLI/VS Code/web).

LangGraph : models the harness as an explicit state graph with two nodes ( llm_call and tool_node) and conditional edges; evolved from LangChain’s AgentExecutor, which was deprecated for poor extensibility.

CrewAI : builds role‑based multi‑agent systems (Agent, Task, Crew) and adds a “deterministic skeleton” in the Flows layer to manage routing and verification.

AutoGen (Microsoft Agent Framework) : introduces a dialogue‑driven orchestration with three layers (Core, AgentChat, Extensions) and five orchestration modes (sequential, concurrent fan‑out/in, group chat, hand‑off, and “magentic” task ledger).

Key Insights and Future Outlook

Empirical evidence shows that changing only the harness can dramatically shift performance: on TerminalBench 2.0, a harness tweak moved a model’s ranking from >30th to 5th, and a research project where the LLM optimized its own infrastructure achieved a 76.4 % success rate, surpassing manually engineered systems.

Despite the trend toward thinner harnesses as models improve, the harness will not disappear. Even the strongest models need mechanisms to manage scarce context, execute tool calls, persist state, and verify work. The “scaffolding” analogy (hardware scaffolding enables workers to reach high places but is removed after construction) captures this: as models become more capable, harness complexity should shrink, but the harness remains the essential layer that makes autonomous agents viable.

When an agent fails, the article advises looking at the harness rather than blaming the model.