Agent Harness Deep Dive: Unpacking the Architecture Behind AI Agents

What is an Agent Harness? The term, coined in early 2026, refers to the complete software infrastructure that wraps a large language model (LLM) to make it behave as an autonomous agent. Anthropic’s Claude Code SDK calls it the "agent harness" that drives Claude Code, while OpenAI’s Codex team treats "agent" and "harness" as synonymous, meaning the non‑model plumbing that makes the model useful. Vivek Trivedy of LangChain defines it succinctly: "If you are not the model itself, you are the harness." Beren Millidge (2023) likens a harness to an operating system: the LLM is a CPU, the context window is RAM, external databases are storage, and the harness provides device drivers, scheduling, and safety checks.

Three concentric engineering layers surround the model. Prompt engineering designs the system prompt and user instructions; context engineering decides what the model sees and when; harness engineering combines the former two with tool orchestration, persistent state, error recovery, verification loops, security guards, and lifecycle management. The harness is therefore more than a prompt wrapper—it is the machinery that enables genuine autonomous behavior.

Production‑grade harness components (derived from Anthropic, OpenAI, LangChain, and community practice) include:

Orchestration loop (the TAO/ReAct cycle)

Tool definitions and sandbox execution

Memory (short‑term dialogue history and long‑term JSON/SQLite stores)

Context management and compression strategies

Prompt construction (system prompt + tool schema + memory + history + user message)

Output parsing (structured tool_calls vs free‑text)

State management (graph‑based reducers, checkpoints, git‑style drafts)

Error handling (retry, LLM‑recoverable errors, user‑fixable errors, unexpected failures)

Safety guards (input, output, and tool‑level guardrails, circuit‑breaker)

Verification loop (rule‑based tests, visual checks, LLM‑as‑judge)

Sub‑agent orchestration (fork, teammate, worktree models)

Deployment policies (single vs multi‑agent, thin vs thick harness)

Full loop walkthrough (seven steps):

Prompt assembly: system prompt, tool schema, memory files, dialogue history, and current user message are concatenated, with high‑signal tokens placed at the start and end (as shown by the "lost in the middle" study).

LLM inference: the assembled prompt is sent to the model API, which returns text, tool calls, or both.

Output classification: if only text is returned, the loop ends; if a tool call appears, execution proceeds; if a hand‑off is signaled, the current agent is swapped.

Tool execution: the harness validates parameters, checks permissions, runs the tool in a sandbox (read‑only calls may run concurrently, writes are serialized), and captures the result.

Result packaging: tool output is formatted as an LLM‑readable observation; errors are returned as error messages for self‑correction.

Context update: the observation is appended to the dialogue history; when the context window nears its limit, the harness triggers compression (e.g., Claude Code keeps architectural decisions and unresolved bugs while discarding redundant tool output).

Loop repeat: the process returns to step 1 until a termination condition is met (no tool call, max rounds, token budget, safety breaker, user interrupt, or refusal).

Key design decisions examined include:

Single‑agent vs multi‑agent: both Anthropic and OpenAI advise perfecting a single agent first; multi‑agent adds routing overhead and context loss, useful only when tool overlap exceeds ~10 or distinct task domains exist.

ReAct vs plan‑execute: ReAct interleaves reasoning and action each step (flexible but costly); plan‑execute separates planning from execution, with LLMCompiler reporting a 3.6× speedup.

Context‑window management: five production strategies (time‑based eviction, dialogue summarization, observation masking, structured notes, sub‑agent delegation). ACON research shows that preserving reasoning traces while discarding raw tool output reduces token usage by 26‑54 % with >95 % accuracy.

Verification loop design: deterministic tests (unit‑style, code checks) give a ground‑truth baseline; LLM‑as‑judge captures semantic issues but adds latency. Martin Fowler’s ThoughtWorks framework separates “guides” (pre‑action hints) from “sensors” (post‑action feedback).

Permission and security architecture: a permissive mode (fast, higher risk) versus a restrictive mode (slow, safer) depending on deployment context.

Tool‑scope strategy: fewer tools improve performance; Vercel removed 80 % of its tools and saw gains; Claude Code uses lazy loading to cut context by 95 %.

Harness thickness: the trade‑off between logic in the harness versus the model. Anthropic bets on a thin harness that shrinks as models improve, while graph‑based frameworks keep explicit control.

Framework implementations illustrate these trade‑offs:

Anthropic Claude Agent SDK exposes a single query() function that runs a "dumb loop" while the model does all reasoning; it follows a collect‑act‑validate cycle.

OpenAI Agents SDK provides a Runner class supporting async, sync, and streaming modes; Codex builds a three‑layer stack (Core + App Server + Client UI) that shares a single harness.

LangGraph models the harness as an explicit state graph with nodes for LLM calls and tool nodes, replacing the deprecated AgentExecutor from LangChain.

CrewAI adds role‑based multi‑agent orchestration (Agent + Task + Crew) with a deterministic routing layer.

AutoGen (evolving into Microsoft Agent Framework) introduces a three‑layer architecture (Core, AgentChat, Extensions) and supports five orchestration patterns: sequential, concurrent, group chat, hand‑off, and magentic task‑ledger management.

Evolution insight – the harness is analogous to construction scaffolding: essential during building but removed after completion. As models improve, harness complexity should shrink; Manus rewrote its harness five times in six months to remove unnecessary layers. Nevertheless, the harness never disappears because even the most capable model still needs context management, tool execution, state persistence, and validation. Empirical evidence: changing only the harness moved a Claude‑based agent from outside the top 30 to rank 5 on TerminalBench 2.0, and an LLM‑self‑optimizing infrastructure achieved a 76.4 % pass rate, surpassing manually engineered systems.

When an agent fails, the article advises looking at the harness rather than blaming the model.