Deep Dive into Agent Harness: Unpacking the Architecture Behind AI Agents

What Is an Agent Harness?

First coined in early 2026, an Agent Harness is the full software stack that surrounds a large language model (LLM) to turn it into a capable autonomous agent. It includes orchestration loops, tool definitions, memory systems, context management, state persistence, error handling, and safety guardrails. Anthropic’s Claude Code SDK explicitly calls its SDK the “agent harness,” and OpenAI’s Codex team treats “agent” and “harness” as synonymous concepts.

Three Engineering Layers

Prompt Engineering : designs the instructions the model receives.

Context Engineering : controls what the model sees and when.

Harness Engineering : combines the first two and adds full application infrastructure (tool orchestration, state persistence, error recovery, validation loops, security, lifecycle management).

12 Core Components of a Production‑Grade Harness

Orchestration Loop : implements the Think‑Act‑Observe (TAO) or ReAct cycle—assemble prompt, call LLM, parse output, invoke tools, feed results back, repeat.

Tools : defined by schema (name, description, parameters) and injected into the LLM context; responsible for registration, validation, sandboxed execution, result capture, and formatting.

Memory : short‑term (conversation history) and long‑term (persistent stores such as Claude Code’s CLAUDE.md, LangGraph’s JSON Store, OpenAI’s SQLite/Redis sessions).

Context Management : mitigates “context decay” where critical information placed in the middle of the window degrades performance by >30 % (Chroma study, corroborated by Stanford’s “Lost in the Middle” paper). Strategies include compression, observation masking, instant retrieval, and sub‑agent delegation.

Prompt Construction : layers system prompt, tool schema, memory files, dialogue history, and current user message. OpenAI’s Codex uses a strict priority stack (system → tools → developer → user).

Output Parsing : modern harnesses rely on structured tool_calls objects; fallback parsers (e.g., RetryWithErrorOutputParser) handle edge cases.

State Management : LangGraph models state as typed dictionaries flowing through graph nodes; checkpoints enable interruption recovery and time‑travel debugging. OpenAI offers four mutually exclusive strategies (in‑process memory, SDK session, Conversations API, previous‑response‑id linking).

Error Handling : each step’s success probability compounds (a 99 % per‑step success over 10 steps yields ~90.4 % end‑to‑end). Harnesses categorize errors as transient, LLM‑recoverable, user‑fixable, or unexpected, with retry limits (e.g., Stripe caps retries at two).

Safety Guardrails : input, output, and tool‑level guardrails; Anthropic separates permission execution from model reasoning, gating ~40 tool capabilities across three phases (trust establishment, per‑call check, high‑risk confirmation).

Verification Loop : rule‑based feedback, visual checks (Playwright screenshots), or LLM‑as‑judge sub‑agents. Claude Code’s creator Boris Cherny reports a 2‑3× quality boost when the model can self‑validate.

Sub‑Agent Orchestration : supports fork, teammate, and worktree execution models (Claude Code) or treats agents as tools and hand‑offs (OpenAI). LangGraph nests sub‑agents as embedded state graphs.

Step‑by‑Step Loop Walk‑through

Prompt Assembly : Harness builds the full input (system prompt + tool schema + memory + history + user message). Critical context is placed at the prompt’s start and end, echoing findings from the “Lost in the Middle” study.

LLM Inference : The assembled prompt is sent to the model API, which returns text, tool calls, or both.

Output Classification : If only text is returned, the loop ends; otherwise, tool execution or agent hand‑off proceeds.

Tool Execution : Parameters are validated, permissions checked, sandboxed execution runs, and results are captured. Read‑only calls may run concurrently; write calls are serialized.

Result Packaging : Tool results are formatted for the LLM; errors are returned as structured error messages for self‑correction.

Context Update : Results are appended to the dialogue history; when nearing the context window limit, the Harness triggers compression.

Loop Continuation : Return to step 1 until a termination condition is met (no tool call, max rounds, token budget exhausted, guardrail trip, user interrupt, or safety refusal).

Framework Implementations

Anthropic Claude Agent SDK : exposes a single query() function that runs a “dumb loop” while the model holds all intelligence. Uses a collect‑act‑validate cycle.

OpenAI Agents SDK : provides a Runner class supporting async, sync, and streaming modes; builds a three‑layer architecture (Core → App Server → Client UI). The shared Harness explains why Codex performs better in its native UI than in generic chat windows.

LangGraph : models the Harness as an explicit state graph with conditional edges (tool call vs. end). Evolved from LangChain’s AgentExecutor, which was deprecated for lack of extensibility.

CrewAI : adds role‑based multi‑agent orchestration (Agent → Task → Crew) with deterministic scaffolding for routing and verification.

AutoGen (Microsoft Agent Framework) : introduces a dialogue‑driven orchestration stack (Core → AgentChat → Extensions) supporting sequential, concurrent, group‑chat, hand‑off, and “magentic” patterns.

Key Design Decisions (Seven Questions)

Single‑agent vs. multi‑agent: start with a robust single agent; split only when tool overlap exceeds ~10 or distinct task domains emerge.

ReAct vs. plan‑execute: ReAct interleaves reasoning and action (flexible but costlier); plan‑execute separates them, with LLMCompiler reporting a 3.6× speedup.

Context‑window management: five production strategies (time‑based eviction, dialogue summarization, observation masking, structured notes, sub‑agent delegation). ACON research shows prioritizing reasoning traces can cut token usage by 26‑54 % while keeping >95 % accuracy.

Verification loop design: deterministic testing (code checks) vs. LLM‑as‑judge (semantic coverage) – each trades latency for coverage (Martin Fowler’s framework).

Permission & security model: “loose” (fast, higher risk) vs. “strict” (slow, safer) depending on deployment context.

Tool‑scope strategy: fewer tools improve performance; Vercel removed 80 % of tools and saw gains; Claude Code’s lazy loading reduces context by 95 %.

Harness thickness: balance how much logic lives in the Harness versus the model. Anthropic bets on thin Harnesses that shrink as models improve, while graph‑based frameworks favor explicit control.

Why the Harness Matters

Benchmark evidence (TerminalBench 2.0) shows that swapping only the Harness can move an agent’s ranking by more than 20 positions, even with the same underlying model. The Harness is therefore the engineering core that manages scarce context, captures failures before they accumulate, provides persistent memory, and enforces safety.

As models become stronger, Harnesses are trending toward thinner designs, but they will not disappear; every powerful model still needs infrastructure to manage its context window, execute tools, persist state, and verify work.

When an agent fails, the fault often lies not in the model but in the surrounding Harness.