Agent Harness Explained: A Deep Dive into AI Agent Architecture

What Is an Agent Harness?

Agent Harness is the complete software stack that surrounds a large language model (LLM) to turn it into a functional agent. It includes the orchestration loop, tool integration, memory systems, context management, state persistence, error handling, and safety guardrails. Anthropic’s Claude Code SDK explicitly calls this infrastructure the “agent harness,” and OpenAI’s Codex team treats “agent” and “harness” as synonymous concepts.

Three Engineering Layers

The architecture can be viewed as three concentric layers around the model:

Prompt Engineering : designs the instructions the model receives.

Context Engineering : decides what the model sees and when, handling window limits and relevance.

Harness Engineering : combines the first two layers with full application infrastructure—tool orchestration, persistent memory, error recovery, verification loops, and security.

12 Core Components of a Production‑Grade Harness

Orchestration Loop (the TAO/ReAct cycle)

Tool Registry and Schema Validation

Memory (short‑term dialogue history and long‑term persistent stores)

Context Management (compression, observation masking, live retrieval, sub‑agent delegation)

Prompt Construction (system prompt, tool definitions, memory files, conversation history, user message)

Output Parsing (structured tool_calls vs free‑form text)

State Management (graph‑based state dictionaries, checkpoints, time‑travel debugging)

Error Handling (retry, LLM‑recoverable errors, user‑fixable errors, unexpected errors)

Safety Guardrails (input, output, and tool‑level checks, circuit‑breaker mechanism)

Verification Loop (rule‑based feedback, visual UI checks, LLM‑as‑evaluator)

Sub‑Agent Orchestration (fork, teammate, worktree models; agent‑as‑tool and hand‑off patterns)

Deployment Strategy (single‑agent vs multi‑agent, ReAct vs plan‑execute, context‑window policies, tool‑scope decisions, harness thickness)

Step‑by‑Step Execution Loop

The production loop proceeds through seven stages:

Prompt Assembly : concatenate system prompt, tool schemas, memory files, conversation history, and the current user message. Critical context is placed at the beginning and end of the prompt (as shown by the “lost in the middle” study).

LLM Inference : send the assembled prompt to the model API; the model returns text, tool calls, or both.

Output Classification : if only text is returned, the loop ends; if a tool call is present, execution proceeds; if a hand‑off is requested, the current agent is swapped and the loop restarts.

Tool Execution : Harness validates parameters, checks permissions, runs the tool in a sandbox, and captures results. Read‑only calls may run concurrently; write calls are serialized.

Result Packaging : tool results are formatted as LLM‑readable messages; errors are returned as error objects for self‑correction.

Context Update : results are appended to the dialogue history; when the context window nears its limit, Harness triggers compression.

Loop Continuation : return to step 1 until a termination condition is met (no tool call, max rounds, token budget exhausted, safety breaker, user interrupt, or security refusal).

Framework Implementations

Major frameworks expose the harness in different ways:

Anthropic Claude Agent SDK : a single query() function creates the loop and returns a streaming iterator; the runtime is a “dumb loop” with intelligence residing in the model.

OpenAI Agents SDK : a Runner class supports async, sync, and streaming modes; the Codex harness adds three layers (Core, App Server, UI) that share the same underlying harness.

LangGraph : models the harness as an explicit state graph with nodes for LLM calls and tool nodes, replacing the deprecated AgentExecutor from LangChain.

CrewAI : builds role‑based multi‑agent systems (Agent, Task, Crew) with a “deterministic skeleton” that routes and validates work.

AutoGen (Microsoft Agent Framework) : provides a three‑layer architecture (Core, AgentChat, Extensions) supporting sequential, concurrent, group‑chat, hand‑off, and magentic orchestration patterns.

Key Design Decisions

Seven recurring decisions shape every harness:

Single vs. Multi‑Agent : start with a single agent; split only when tool overlap exceeds ~10 or distinct task domains emerge.

ReAct vs. Plan‑Execute : ReAct interleaves reasoning and action (flexible but costly); plan‑execute separates planning from execution (LLMCompiler reports 3.6× speedup).

Context‑Window Management : five production strategies—time‑based eviction, dialogue summarization, observation masking, structured notes, sub‑agent delegation. ACON research shows prioritizing reasoning traces can cut token usage by 26‑54 % while keeping >95 % accuracy.

Verification Loop Design : deterministic rule‑based checks (tests, static analysis) vs. LLM‑as‑evaluator (captures semantic bugs but adds latency). Martin Fowler’s framework distinguishes “guides” (pre‑action) and “sensors” (post‑action).

Permission & Safety Model : “lenient” (fast, higher risk) vs. “restrictive” (slow, higher assurance) depending on deployment context.

Tool‑Scope Strategy : fewer tools improve performance; Vercel removed 80 % of tools and saw gains; Claude Code uses lazy loading to cut context by 95 %.

Harness Thickness : balance logic between harness and model. Anthropic bets on thin harnesses that shrink as models improve, while graph‑based frameworks keep explicit control.

Why Harness Matters

Even with the strongest LLMs, production performance hinges on the harness. TerminalBench 2.0 showed that swapping only the harness moved an agent from outside the top 30 to rank 5. Harnesses manage scarce context, enforce verification before error accumulation, provide persistent memory, and mediate tool execution and safety—functions that no model can replace.

Conclusion

Agent Harness is not a solved problem or a commoditized layer; it is the core engineering discipline that determines whether an AI agent can operate reliably at scale. As models evolve, harnesses will become thinner, but they will never disappear.