Deep Dive into Agent Harness: Dissecting the Architecture Behind AI Agents

What Is an Agent Harness

Agent Harness is the complete software layer that wraps a large language model (LLM) to turn it into a functional agent. It provides orchestration loops, tool definitions, memory systems, context management, state persistence, error handling, safety guardrails, and validation cycles. Anthropic’s Claude Code SDK and OpenAI’s Codex SDK both refer to this layer as the “agent harness,” emphasizing that the model itself is unchanged while the surrounding infrastructure drives capability.

Key Distinction: Agent vs. Harness

According to Vivek Trivedy of LangChain, if you are not the model, you are the harness. The agent is the emergent behavior—goal‑directed, tool‑using, self‑correcting entity—whereas the harness is the machinery that enables that behavior. Beren Millidge (2023) likens a bare LLM to a CPU without memory, storage, or I/O, and the harness to an operating system that makes the CPU useful.

Three Engineering Layers

Prompt Engineering : designs the instructions the model receives.

Context Engineering : decides what the model sees and when, preventing context‑window decay (e.g., “lost in the middle” phenomenon where performance drops >30% when key information is mid‑window).

Harness Engineering : combines the above with tool orchestration, persistent memory, error recovery, verification loops, and security.

Production‑Grade Harness Components

A synthesis of Anthropic, OpenAI, LangChain, and community practice identifies twelve essential components:

Orchestration Loop : Implements the ReAct (think‑act‑observe) cycle by assembling prompts, invoking the LLM, parsing output, executing tools, and feeding results back.

Tools : Defined via schemas (name, description, parameters) and injected into the LLM context. Claude Code offers six tool categories; OpenAI’s Agents SDK supports function tools, hosted tools, and MCP server tools.

Memory : Short‑term (conversation history) and long‑term (persistent stores such as Claude Code’s CLAUDE.md, LangGraph’s JSON Store, OpenAI’s SQLite/Redis sessions).

Context Management : Uses compression, observation masking, and on‑demand retrieval to keep the token budget low while preserving high‑signal information.

Prompt Construction : Stacks system prompts, tool schemas, memory files, dialogue history, and user messages, placing critical context at the start and end.

Output Parsing : Modern harnesses rely on structured tool_calls objects; fallback parsers (e.g., RetryWithErrorOutputParser) handle free‑text failures.

State Management : LangGraph models state as typed dictionaries flowing through graph nodes; checkpoints enable interruption recovery and time‑travel debugging. OpenAI offers four mutually exclusive strategies (in‑memory, SDK session, Conversations API, previous‑response linking).

Error Handling : Classifies errors as transient, LLM‑recoverable, user‑fixable, or unexpected. A 99% per‑step success rate yields ~90% end‑to‑end success for a ten‑step workflow.

Safety Guardrails : Input, output, and tool guardrails; a “circuit‑breaker” aborts the agent when triggered. Anthropic separates permission checks from model reasoning.

Verification Loop : Rule‑based checks, visual UI validation (Playwright screenshots), or LLM‑as‑judge sub‑agents assess outputs. Claude Code’s creator Boris Cherny reports a 2‑3× quality boost from self‑verification.

Sub‑Agent Orchestration : Supports forked contexts, teammate panels, or independent git worktrees (Claude Code). OpenAI treats sub‑agents as tools or hand‑offs; LangGraph nests state graphs.

Step‑by‑Step Agent Cycle

The article walks through a full loop:

Prompt Assembly : Combine system prompt, tool schemas, memory, history, and user input.

LLM Inference : Model returns text, tool calls, or both.

Output Classification : If only text, terminate; if tool calls, execute; if hand‑off, switch agents.

Tool Execution : Validate parameters, check permissions, run in sandbox, capture results (read‑only can run concurrently).

Result Packaging : Format tool output for the LLM; errors are returned as structured observations.

Context Update : Append results to history; trigger compression near token limits.

Loop Continuation : Repeat until termination conditions (no tool call, max rounds, token budget, safety break, user interrupt, or refusal).

Design Decisions and Trade‑offs

Key decisions include single‑ vs. multi‑agent architectures (single‑agent preferred until tool overlap exceeds ~10), ReAct vs. plan‑execute (plan‑execute can be 3.6× faster per LLMCompiler), context‑window strategies (time‑based eviction, summarization, observation masking, structured notes, sub‑agent delegation), validation approaches (rule‑based vs. LLM‑as‑judge), and security models (lenient vs. restrictive). Studies cited: ACON research on token savings, Martin Fowler’s Thoughtworks framework for guidelines and sensors.

Why Harness Matters

Empirical evidence from TerminalBench 2.0 shows that swapping only the harness can move an agent’s ranking from outside the top 30 to #5, demonstrating that harness design can outweigh raw model improvements. As models improve, harnesses are expected to become thinner, but they will never disappear because they manage scarce context, tool execution, state persistence, and verification.