Deep Dive into Agent Harness: Dissecting the Architecture Behind AI Agents

What Is an Agent Harness?

Agent Harness is the complete software stack that wraps a large language model (LLM) to make it behave as an autonomous agent. It includes the reasoning‑action‑observation (ReAct) loop, tool definitions, memory management, context handling, state persistence, error handling, safety guardrails, and lifecycle management. Anthropic’s Claude Code SDK explicitly calls this "the agent harness" and OpenAI’s Codex SDK treats "agent" and "harness" as synonymous concepts.

Three Engineering Layers

Prompt Engineering : designs the instructions the model receives.

Context Engineering : decides what the model sees and when, preventing context‑window degradation.

Harness Engineering : combines the first two layers with tool orchestration, persistent memory, error recovery, verification loops, security guardrails, and lifecycle control.

Production‑Grade Harness: Twelve Core Components

Orchestration Loop : implements the TAO (think‑act‑observe) cycle, often a simple while loop whose complexity lies in the surrounding management.

Tools : schema‑defined utilities (e.g., file I/O, web search, code execution) injected into the LLM’s context; Anthropic provides six categories, OpenAI supports function tools and hosted tools.

Memory : short‑term dialogue history and long‑term persistent stores (Claude Code uses CLAUDE.md and MEMORY.md, LangGraph uses namespaced JSON stores, OpenAI can back memory with SQLite or Redis).

Context Management : mitigates "context decay" (e.g., a 30 % drop when key information sits in the middle of the window, as shown by Chroma and the Stanford "Lost in the Middle" paper). Strategies include compression, observation masking, and just‑in‑time retrieval.

Prompt Construction : assembles system prompts, tool schemas, memory files, dialogue history, and the current user message, placing critical context at the start and end.

Output Parsing : modern harnesses expect structured tool_calls objects; if absent, the output is treated as the final answer.

State Management : models state as a typed dictionary flowing through graph nodes (LangGraph) or as SDK sessions (OpenAI) and supports checkpointing for interruption recovery.

Error Handling : classifies errors (transient, LLM‑recoverable, user‑fixable, unexpected) and applies retries or circuit‑breaker termination; Stripe’s production harness caps retries at two.

Guardrails & Security : three‑tier guardrails (input, output, tool) plus a circuit‑breaker; Anthropic separates permission checks from model reasoning, gating ~40 tools in three phases.

Verification Loop : validates work via rule‑based tests, visual checks (Playwright screenshots), or LLM‑as‑judge sub‑agents; Claude Code’s creator Boris Cherny reports a 2‑3× quality boost when the model self‑verifies.

Sub‑Agent Orchestration : supports forked execution, teammate panels, or git‑based worktrees (Claude Code) and allows expert‑as‑tool or expert‑as‑handoff patterns (OpenAI, LangGraph).

Step‑by‑Step Loop Execution

The article walks through a full cycle:

Prompt Assembly : combine system prompt, tool schemas, memory, history, and user input; important context is placed at the boundaries.

LLM Inference : send the assembled prompt to the model API; the model returns text, tool calls, or both.

Output Classification : if only text is returned, the loop ends; if a tool call appears, proceed to execution; if a handoff is indicated, switch the active agent and restart.

Tool Execution : validate parameters, check permissions, run in a sandbox, capture results; read‑only calls may run concurrently, writes are serialized.

Result Packaging : format tool results as LLM‑readable messages; errors are returned as error results for self‑correction.

Context Update : append results to dialogue history; trigger compression when approaching the context window limit.

Loop Continuation : return to step 1 until a termination condition is met (no tool call, max rounds, token budget, guardrail trip, user interrupt, or safety refusal).

Framework Implementations

Anthropic’s Claude Agent SDK exposes a single query() function that runs a "dumb loop" where all intelligence resides in the model. OpenAI’s Agents SDK uses a Runner class supporting async, sync, and streaming modes; Codex builds three layers (core, app server, UI) that share a single harness. LangGraph models the harness as an explicit state graph, replacing the deprecated AgentExecutor. CrewAI adds role‑based multi‑agent orchestration, while AutoGen (now Microsoft Agent Framework) introduces three‑layer orchestration (Core, AgentChat, Extensions) with five execution patterns.

Key Design Decisions (Seven Questions)

Single‑agent vs. multi‑agent: start with a strong single agent; split only when tool overlap exceeds ~10 or distinct task domains emerge.

ReAct vs. plan‑execute: ReAct interleaves reasoning and action (flexible but costly); plan‑execute separates planning, yielding up to 3.6× speedups (LLMCompiler).

Context‑window strategy: time‑based eviction, dialogue summarization, observation masking, structured notes, sub‑agent delegation; ACON research shows 26‑54 % token savings while keeping >95 % accuracy.

Verification loop design: deterministic test suites vs. LLM‑as‑judge; the former offers ground‑truth guarantees, the latter captures semantic issues at the cost of latency.

Permission model: permissive (fast, riskier) vs. restrictive (slow, safer) depending on deployment.

Tool‑scope policy: fewer tools improve performance; Vercel removed 80 % of tools and saw gains; lazy loading can cut context usage by 95 %.

Harness thickness: balance how much logic lives in the harness versus the model; Anthropic prefers a thin harness that shrinks as models improve.

Harness as a Product

Changing only the harness can shift an agent’s ranking on benchmarks like TerminalBench by more than 20 positions, demonstrating that harness design is a product‑level differentiator. The harness must manage scarce context, capture failures early, provide reliable memory, and decide how much responsibility to offload to the model.

Conclusion

When a production agent fails, the culprit is rarely the LLM itself; it is often the surrounding harness. Understanding and engineering the harness—its loops, tools, memory, context policies, error handling, and safety layers—is therefore essential for reliable, scalable AI agents.