Deep Dive into Agent Harness: Dissecting the Architecture of AI Agents

What Is an Agent Harness?

Agent Harness is the complete software infrastructure that wraps a large language model (LLM) to make it a usable, autonomous agent. It includes the orchestration loop, tool integration, memory, context management, state persistence, error handling, and safety guardrails. Anthropic’s Claude Code SDK explicitly calls this the "agent harness," and OpenAI’s Codex SDK treats "agent" and "harness" as synonymous concepts.

Three Engineering Layers

Prompt Engineering : designs the instructions the model receives.

Context Engineering : controls what the model sees and when, preventing context decay.

Harness Engineering : combines the first two layers with full application infrastructure (tool orchestration, state persistence, error recovery, verification loops, and security).

Production‑Grade Harness: Twelve Components

Orchestration Loop – Implements the TAO (think‑act‑observe) or ReAct cycle: assemble prompt, call LLM, parse output, execute tool, feed result back, repeat.

Tools – Defined by schema (name, description, parameters) and injected into the LLM context; responsible for registration, validation, sandboxed execution, and result formatting. Anthropic provides six tool categories; OpenAI supports function tools, hosted tools, and MCP server tools.

Memory – Short‑term (conversation history) and long‑term (persistent stores). Claude Code uses lightweight indexes, on‑demand detailed files, and raw records; LangGraph uses namespaced JSON stores; OpenAI can back sessions with SQLite or Redis.

Context Management – Mitigates "context decay" (performance drop >30 % when key content sits in the middle of the window, as shown by the Chroma study). Strategies include compression, observation masking, instant retrieval, and sub‑agent delegation.

Prompt Construction – Stacks system prompt, tool schemas, memory files, conversation history, and the current user message. OpenAI’s Codex prioritises system messages, tool definitions, developer instructions, and user input (32 KiB limit).

Output Parsing – Modern harnesses rely on structured tool_calls objects; if no tool call is present, the response is final. Pydantic schemas can constrain responses; legacy parsers like RetryWithErrorOutputParser remain for edge cases.

State Management – LangGraph models state as typed dictionaries flowing through graph nodes; checkpoints enable interruption recovery and time‑travel debugging. OpenAI offers four mutually exclusive strategies (in‑app memory, SDK session, Conversations API, previous‑response linking). Claude Code uses git commits as checkpoints.

Error Handling – Errors compound quickly (a 99 % per‑step success rate over ten steps yields ~90 % end‑to‑end success). Harnesses classify errors as transient, LLM‑recoverable, user‑fixable, or unexpected, applying retries, tool‑level guardrails, or circuit‑breaker stops.

Guardrails & Security – Input, output, and tool guardrails run at different layers. Anthropic separates permission execution from model reasoning, gating ~40 tool capabilities through staged checks and explicit user confirmation for high‑risk actions.

Verification Loop – Production agents use rule‑based feedback, visual checks (e.g., Playwright screenshots), or an LLM evaluator sub‑agent. Claude Code’s creator Boris Cherny reports a 2‑3× quality boost when the model can self‑verify.

Sub‑Agent Orchestration – Supports fork (byte‑level copy), teammate (independent terminal panels with file‑based mailbox), and worktree (isolated git branches). OpenAI allows agents as tools or hand‑offs; LangGraph nests sub‑agents as state graphs.

Decision Framework – Seven recurring decisions: single vs. multi‑agent, ReAct vs. plan‑execute, context‑window strategy, verification design, permission model, tool‑scope policy, and harness thickness (how much logic lives in the harness vs. the model).

Step‑by‑Step Loop Execution

Prompt assembly: combine system prompt, tool schemas, memory files, conversation history, and user input; place critical context at the start and end.

LLM inference: send the assembled prompt to the model API; receive text, tool calls, or both.

Output classification: if only text, terminate; if tool calls, proceed to execution; if hand‑off, switch agents and restart.

Tool execution: validate parameters, check permissions, run in a sandbox, capture results; read‑only calls may run concurrently, writes are serialized.

Result packaging: format tool output for the LLM; capture errors as error results for self‑correction.

Context update: append results to history; trigger compression when approaching window limits.

Loop continuation: repeat from step 1 until a termination condition (no tool call, max rounds, token budget, guard‑rail trigger, user interrupt, or safety refusal) is met.

Framework Implementations

Anthropic’s Claude Agent SDK exposes the harness via a single query() async iterator, implementing a "dumb loop" where intelligence stays in the model. OpenAI’s Agents SDK provides a Runner class supporting async, sync, and streaming modes; Codex builds a three‑layer stack (core, app server, client UI) that shares a single harness. LangGraph models the harness as an explicit state graph, evolving from LangChain’s AgentExecutor. CrewAI adds role‑based multi‑agent orchestration, while AutoGen (Microsoft Agent Framework) introduces three‑layer architecture (Core, AgentChat, Extensions) with five orchestration patterns (sequential, concurrent, group chat, hand‑off, and magentic).

Key Insights and Evolution

The harness is analogous to a construction scaffold: it enables access to otherwise unreachable positions but is removed once the building is complete. As models improve, harness complexity should shrink—e.g., Vercel removed 80 % of its tools and saw performance gains, and Claude Code lazily loads tools to cut context usage by 95 %.

Design for the future: a well‑engineered harness should allow performance gains from stronger models without adding complexity. Even the most capable models still need a harness to manage context, execute tools, persist state, and verify work.

Conclusion

When an agent fails, the blame often lies not with the model but with the surrounding harness. Understanding and engineering the harness is essential for building reliable, production‑grade AI agents.