Agent Harness Unpacked: A Deep Dive into AI Agent Architecture

What Is an Agent Harness?

An Agent Harness is the complete software stack that surrounds a large language model (LLM) to enable autonomous behavior. It includes the orchestration loop, tool integration, memory, context management, state persistence, error handling, and safety guardrails. Anthropic’s Claude Code SDK explicitly calls this stack the “agent harness,” and OpenAI’s Codex SDK treats agent and harness as synonymous.

Vivek Trivedy of LangChain defines it succinctly: “If you are not the model itself, you are the harness.” The distinction is that the agent is the emergent goal‑directed entity, while the harness provides the machinery that makes the agent possible.

Three Engineering Layers

Prompt Engineering : designing the instructions the model receives.

Context Engineering : deciding what the model sees and when, mitigating “context decay” (a >30% drop in performance when key information sits in the middle of the window, as shown by the Chroma study and the Stanford “Lost in the Middle” paper).

Harness Engineering : the full application infrastructure that combines the first two layers with tool orchestration, state persistence, error recovery, verification loops, and security.

12 Core Components of a Production‑Grade Harness

Orchestration Loop : the heartbeat that implements the Think‑Act‑Observe (TAO) or ReAct cycle—assemble prompts, call the LLM, parse output, execute tools, feed results back, and repeat.

Tools : defined by schema (name, description, parameters) and injected into the LLM context. Anthropic offers six categories (file ops, search, execution, web access, code intelligence, sub‑agent generation); OpenAI’s Agents SDK supports function tools, hosted tools, and MCP server tools.

Memory : short‑term (conversation history) and long‑term (persistent stores). Claude Code uses lightweight indexes, on‑demand detailed files, and raw records; LangGraph uses namespaced JSON stores; OpenAI can back sessions with SQLite or Redis.

Context Management : strategies such as compression (summarizing near‑limit history), observation masking (e.g., JetBrains Junie hides old tool output), instant retrieval (lazy loading via grep/glob), and sub‑agent delegation (returning 1‑2 k token summaries).

Prompt Construction : layered assembly of system prompt, tool schema, memory files, dialogue history, and user message. OpenAI’s Codex prioritises system messages, then tool definitions, developer instructions, and finally user input (32 KiB limit).

Output Parsing : modern harnesses rely on structured tool_calls objects rather than free‑text parsing. Both OpenAI and LangChain can enforce schemas with Pydantic; legacy parsers like RetryWithErrorOutputParser remain for edge cases.

State Management : LangGraph models state as a typed dictionary flowing through graph nodes with reducers; checkpoints enable interruption recovery. OpenAI offers four mutually exclusive strategies (in‑app memory, SDK session, Conversations API, lightweight previous_response_id linking). Claude Code uses git commits as checkpoints.

Error Handling : each step’s 99% success rate compounds—ten steps yield ~90.4% end‑to‑end success. Harnesses classify errors into transient (with back‑off), LLM‑recoverable (returned as tool messages), user‑fixable, and unexpected (bubbled up). Anthropic returns error objects from tool handlers; Stripe’s production harness caps retries at two.

Safety Guardrails : input, output, and tool‑level guards. OpenAI’s SDK implements a three‑layer guard and a circuit‑breaker that halts the agent on violation. Anthropic separates permission execution from model reasoning, gating ~40 tools across three phases (trust build, per‑call check, high‑risk confirmation).

Verification Loop : production agents use rule‑based feedback, visual checks (Playwright screenshots), or LLM‑as‑judge sub‑agents. Claude Code’s creator Boris Cherny reports a 2–3× quality boost when the model can self‑verify.

Sub‑Agent Orchestration : frameworks differ—Claude Code supports Fork (byte‑level copy), Teammate (file‑based mailbox), and Worktree (git branches); OpenAI treats sub‑agents as tools or hand‑offs; LangGraph nests sub‑agents as state graphs.

Step‑by‑Step Loop Walk‑Through

Prompt Assembly : combine system prompt, tool schemas, memory files, history, and user query. Critical context is placed at the start and end, following the “lost in the middle” insight.

LLM Inference : send the assembled prompt to the model API; receive text, tool calls, or both.

Output Classification : if only text, terminate; if tool calls, proceed; if a hand‑off is indicated, switch the active agent and restart.

Tool Execution : validate parameters, check permissions, run in a sandbox, capture results. Read‑only calls may run concurrently; write calls are serialized.

Result Packaging : format tool results as LLM‑readable messages; capture errors for self‑correction.

Context Update : append results to history; trigger compression when approaching token limits.

Loop Continuation : return to step 1 until a termination condition is met (no tool call, max rounds, token budget, guard‑breaker, user interrupt, or safety refusal).

Framework Implementations

Anthropic’s Claude Agent SDK exposes the harness via a single query() async iterator, running a “dumb loop” where intelligence lives in the model. OpenAI’s Agents SDK uses a Runner class supporting async, sync, and streaming modes; Codex builds a three‑layer stack (Core, App Server, UI) that shares a single harness, explaining why Codex‑driven UI outperforms generic chat windows.

LangGraph models the harness as an explicit state graph, replacing the deprecated AgentExecutor from LangChain (v0.2) to enable multi‑agent support. CrewAI adds role‑based multi‑agent orchestration (Agent, Task, Crew) with a “deterministic skeleton” for routing and verification. AutoGen (Microsoft Agent Framework) introduces a three‑layer architecture (Core, AgentChat, Extensions) and five orchestration patterns: sequential, concurrent (fan‑out/fan‑in), group chat, hand‑off, and “magentic” task bookkeeping.

Key Decision Points and Trade‑offs

Single vs. Multi‑Agent : start with a single agent; split only when >10 overlapping tools or clearly separate task domains exist (Anthropic, OpenAI recommendation).

ReAct vs. Plan‑Execute : ReAct interleaves reasoning and action (flexible but costly); Plan‑Execute separates planning from execution (LLMCompiler reports 3.6× speedup).

Context Window Management : five production strategies—time‑based eviction, dialogue summarisation, observation masking, structured notes, sub‑agent delegation. ACON research shows 26‑54% token reduction while keeping >95% accuracy by preserving reasoning traces instead of raw tool output.

Verification Design : deterministic tests (code checks) give ground‑truth benchmarks; LLM‑as‑judge captures semantic issues but adds latency (Martin Fowler’s Thoughtworks framework: guides vs. sensors).

Permission Model : “lenient” (fast, higher risk) vs. “restrictive” (slow, safer) choices depend on deployment context.

Tool Scope : more tools often degrade performance; Vercel removed 80% of tools and saw improvements; Claude Code’s lazy loading cuts context usage by 95%.

Harness Thickness : balance logic in the harness vs. in the model. Anthropic bets on thinner harnesses as models improve; graph‑based frameworks keep explicit control.

Empirical Evidence

Changing only the harness moved a system from >30th place to 5th on TerminalBench 2.0 (Anthropic benchmark). A separate research project let the LLM optimise its own infrastructure, achieving a 76.4% pass rate—outperforming manually designed systems. Error‑accumulation analysis shows a 99% per‑step success rate yields ~90.4% overall success for a 10‑step workflow.

These results illustrate that the harness, not the model, is often the decisive factor in production‑grade agent performance.

Conclusion

The Agent Harness is the indispensable, evolving scaffold that enables LLMs to act autonomously. While model capabilities continue to rise, a well‑engineered harness—managing context, tools, state, verification, and safety—remains essential. When an agent fails, the first place to look is the harness, not the model.