What Is an Agent Harness? A Deep Dive into AI Agent Architecture

What Is an Agent Harness?

Agent Harness is the complete software stack that wraps a large language model (LLM) to make it behave as a useful, autonomous agent. It includes orchestration loops, tool integration, memory, context management, state persistence, error handling, and safety guardrails. Anthropic’s Claude Code SDK explicitly calls this stack the "agent harness," and OpenAI’s Codex team treats "agent" and "harness" as synonymous concepts.

Three Engineering Layers

The architecture can be viewed as three concentric layers around the model:

Prompt‑engineering : designs the instructions the model receives.

Context‑engineering : decides what the model sees and when.

Harness‑engineering : combines the first two layers with full application infrastructure—tool orchestration, state persistence, error recovery, verification loops, and security.

The harness is not merely a wrapper for prompts; it is the machinery that enables autonomous agent behavior.

12 Core Components of a Production‑Grade Harness

Orchestration Loop : Implements the think‑act‑observe (TAO) or ReAct cycle—assemble prompt, call LLM, parse output, execute tool, feed result back, repeat.

Tools : Defined by schema (name, description, parameters) and injected into the LLM context. Anthropic provides six tool categories; OpenAI supports function tools, hosted tools, and MCP server tools.

Memory : Short‑term (conversation history) and long‑term (persistent stores such as Claude Code’s .md files, LangGraph JSON namespaces, SQLite/Redis sessions).

Context Management : Prevents context‑window decay (e.g., >30 % performance drop when key content sits in the middle of the window, as shown by Chroma research). Strategies include compression, observation masking, and on‑demand retrieval.

Prompt Construction : Stacks system prompt, tool schema, memory files, dialogue history, and current user message, placing critical context at the beginning and end.

Output Parsing : Modern harnesses rely on structured tool_calls objects; if no tool call is present, the response is final.

State Management : LangGraph models state as a typed dictionary flowing through graph nodes; checkpoints enable interruption recovery and time‑travel debugging. OpenAI offers four mutually exclusive strategies (in‑memory, SDK session, Conversations API, previous‑response‑id linking).

Error Handling : Errors compound quickly (a 99 % per‑step success rate yields ~90 % end‑to‑end success for a 10‑step flow). Harnesses classify errors into transient, LLM‑recoverable, user‑fixable, and unexpected, applying retries or circuit‑breakers accordingly.

Safety Guardrails : Input, output, and tool guardrails run at different layers; Anthropic separates permission execution from model reasoning, using a multi‑stage approval process.

Verification Loop : Production agents use rule‑based tests, visual checks (e.g., Playwright screenshots), or LLM‑as‑evaluator sub‑agents. Claude Code’s creator reports a 2‑3× quality boost when the model can verify its own work.

Sub‑Agent Orchestration : Supports forked execution, teammate panels, or git‑worktree isolation. Frameworks such as CrewAI and AutoGen add role‑based multi‑agent routing and dynamic task delegation.

Step‑by‑Step Loop Execution

The full cycle proceeds as follows:

Prompt Assembly : Harness builds the full input (system prompt + tool schema + memory + history + user message). Critical context is placed at the prompt’s head and tail.

LLM Inference : The assembled prompt is sent to the model API, which returns text, tool calls, or both.

Output Classification : If only text is returned, the loop ends; if a tool call is present, execution proceeds; if a hand‑off is requested, the current agent is swapped.

Tool Execution : Parameters are validated, permissions checked, and the tool runs in a sandbox. Read‑only calls may run concurrently; write calls are serialized.

Result Packaging : Tool results are formatted as LLM‑readable messages; errors are captured and returned for self‑correction.

Context Update : Results are appended to the dialogue history; when the context window nears its limit, the harness triggers compression.

Loop Continuation : Return to step 1 until a termination condition is met (no tool call, max rounds, token budget exhausted, guardrail trip, user interrupt, or safety rejection).

Framework Implementations

Major open‑source and vendor frameworks embody the harness concept:

Anthropic Claude Agent SDK : Exposes a single query() function that runs a "dumb loop" while the model performs all intelligence. Uses a collect‑act‑validate cycle.

OpenAI Agents SDK : Provides a Runner class supporting async, sync, and streaming modes. Codex builds three layers (Core, App Server, UI) on top of this runner.

LangGraph : Models the harness as an explicit state graph, evolving from LangChain’s AgentExecutor. Uses conditional edges to route to tool nodes or termination.

CrewAI : Introduces role‑based multi‑agent orchestration (Agent + Task + Crew) with a "deterministic skeleton" for routing and verification.

AutoGen (Microsoft Agent Framework) : Offers three‑tier architecture (Core, AgentChat, Extensions) and five orchestration modes (sequential, concurrent, group chat, hand‑off, magentic).

Seven Design Decisions Every Harness Must Make

Single vs. Multi‑Agent : Both Anthropic and OpenAI recommend perfecting a single agent first; multi‑agent adds routing overhead and context loss. Split only when tool overlap exceeds ~10 or distinct task domains exist.

ReAct vs. Plan‑Execute : ReAct interleaves reasoning and action each step (flexible but costly). Plan‑Execute separates planning from execution; LLMCompiler reports a 3.6× speedup over sequential ReAct.

Context‑Window Management : Five production strategies—time‑based eviction, dialogue summarization, observation masking, structured notes, sub‑agent delegation. ACON research shows preserving reasoning traces while discarding raw tool output cuts token usage by 26‑54 % with >95 % accuracy.

Verification Loop Design : Deterministic tests (unit, type checks) give a ground‑truth baseline; LLM‑as‑evaluator captures semantic issues but adds latency. Martin Fowler frames this as “guides” (pre‑action) and “sensors” (post‑action).

Permission & Security Model : Choose between permissive (fast, higher risk) and restrictive (slow, safer) guardrails based on deployment context.

Tool‑Scope Strategy : Fewer tools improve performance; Vercel removed 80 % of tools and saw gains. Lazy loading (Claude Code) reduces context by 95 %.

Harness Thickness : Decides how much logic resides in the harness versus the model. Anthropic favors a thin harness that shrinks as models improve; graph‑based frameworks keep more explicit control.

Conclusion

Even when the underlying LLM is identical, changing only the harness can shift an agent’s ranking on benchmarks like TerminalBench by more than twenty positions. The harness is the hard‑core engineering layer that manages scarce context, captures failures early, provides persistent memory, and enforces safety. As models become more capable, harnesses will become thinner, but they will never disappear—every production‑grade AI agent still needs a harness.