Deep Dive into Agent Harness: Turning LLM Failures into Robust AI Agents

What is Agent Harness?

Agent Harness is the complete software infrastructure that wraps a large language model (LLM), providing orchestration loops, tools, memory, context management, state persistence, error handling, and guardrails. Anthropic and OpenAI describe their SDKs as the "Agent Harness" that makes LLMs truly useful beyond the model itself.

Three Engineering Layers

Prompt Engineering : carefully crafted instructions fed to the model.

Context Engineering : managing what the model sees and when it sees it.

Harness Engineering : combines the above and adds tool orchestration, state persistence, error recovery, validation loops, security, and lifecycle management.

The 12 Components of a Production‑grade Harness

1. Orchestration Loop

The core "think‑act‑observe" (TAO) or ReAct loop assembles prompts, calls the LLM, parses output, executes tool calls, feeds results back, and repeats until completion. In practice it is a simple while loop; complexity lies in what the loop manages.

2. Tools

Tools are the agent’s hands. They are defined by a schema (name, description, parameter types) and injected into the LLM context. The tool layer handles registration, schema validation, argument extraction, sandboxed execution, result capture, and formatting for the model.

Claude Code offers six tool categories: file operations, search, execution, web access, code intelligence, and sub‑agent spawning. OpenAI’s Agents SDK supports function tools via @function_tool, hosted tools (WebSearch, CodeInterpreter, FileSearch), and MCP server tools.

3. Memory

Memory provides temporal behavior. Short‑term memory stores the dialogue history of a single session; long‑term memory persists across sessions. Anthropic uses CLAUDE.md and auto‑generated MEMORY.md files; LangGraph uses a JSON Store organized by namespace; OpenAI supports SQLite or Redis back‑ends.

Claude Code implements a three‑level hierarchy: lightweight index (≈150 characters per entry, always loaded), on‑demand detailed topic files, and raw conversation logs accessed only via search.

4. Context Management

Context rot occurs when critical content sits in the middle of the window, degrading model performance by over 30 % (Chroma study, Stanford "Lost in the Middle"). Production strategies include:

Compaction : summarise dialogue when approaching limits (Claude Code keeps architecture decisions and unresolved bugs, discarding redundant tool output).

Observation masking : hide old tool output while keeping tool calls visible (JetBrains Junie).

Just‑in‑time retrieval : maintain lightweight identifiers and load data on demand using commands such as grep, glob, head, tail (Claude Code).

Sub‑agent delegation : each sub‑agent returns a compressed summary of 1 000–2 000 tokens.

Anthropic’s guide aims to find the smallest high‑signal token set to maximise expected results.

5. Prompt Construction

Prompt construction assembles system prompts, tool schemas, memory files, dialogue history, and the current user message. OpenAI’s Codex uses a strict priority stack: server‑controlled system messages, tool definitions, developer instructions, user instructions (via AGENTS.md, 32 KB limit), then dialogue history.

6. Output Parsing

Modern harnesses rely on native tool‑call objects ( tool_calls) rather than free‑form text. The harness checks for tool calls; if present, it executes them and loops, otherwise it returns the final answer. Structured output can be constrained with Pydantic models, e.g., RetryWithErrorOutputParser, which feeds the original prompt, failed completion, and parsing error back to the model.

7. State Management

LangGraph models state as a typed dictionary flowing through graph nodes, merged via reducers. Checkpoints are generated at super‑step boundaries, enabling interruption recovery and time‑travel debugging. OpenAI offers four mutually exclusive strategies: application‑level memory, SDK Session, server‑side Conversations API, or lightweight chaining via previous_response_id. Claude Code uses git commits as checkpoints and progress files as temporary workspaces.

8. Error Handling

In a ten‑step process with 99 % per‑step success, end‑to‑end success is only ~90.4 %; errors compound quickly. LangGraph distinguishes four error types: transient (retry with back‑off), LLM‑recoverable (return a ToolMessage for the model to adjust), user‑recoverable (pause for human input), and unexpected (bubble up for debugging). Anthropic captures tool failures internally and returns them as error results to keep the loop running. Stripe’s production harness caps retries at two.

9. Guardrails & Security

OpenAI’s SDK implements three guardrail layers: input guardrails on the first agent, output guardrails on the final answer, and tool guardrails on each tool call, with a tripwire that stops the agent immediately. Anthropic separates permission execution from model reasoning, applying three stages of trust: load‑time trust establishment, per‑call permission checks, and explicit user confirmation for high‑risk operations.

10. Validation Loop

Validation distinguishes production agents from demos. Anthropic recommends rule‑based feedback (tests, type checkers), visual feedback (Playwright screenshots), and LLM‑as‑judge (sub‑agent evaluation). Claude Code’s creator reports a 2‑3× quality boost when models receive validation of their work.

11. Sub‑Agent Orchestration

Claude Code supports three execution modes: Fork (byte‑level copy of parent context), Teammate (independent terminal panel with file‑based mailbox communication), and Worktree (isolated git worktree per agent). OpenAI’s SDK supports "agent‑as‑tool" (expert agents handling well‑defined sub‑tasks) and "handoff" (expert agents taking full control). LangGraph implements sub‑agents as nested state graphs.

12. Loop Execution Details

Step 1 – Prompt Assembly: Harness builds the full input (system prompt, tool schema, memory files, dialogue history, user message) with important context placed at the start and end, following the "Lost in the Middle" principle.

Step 2 – LLM Inference: The assembled prompt is sent via API; the model returns tokens that may include text, tool‑call requests, or both.

Step 3 – Output Classification: If only text is produced, the loop ends; if a tool call is requested, execution proceeds; if a handoff is requested, the current agent is updated and the loop restarts.

Step 4 – Tool Execution: Each tool call is validated, permission‑checked, sandboxed, and its result captured. Read‑only calls may run concurrently; mutating calls run serially.

Step 5 – Result Packaging: Tool results are formatted as LLM‑readable messages; errors are returned as error results for self‑correction.

Step 6 – Context Update: The final result is appended to dialogue history; if near the token limit, compression is triggered.

Step 7 – Loop Continuation: Return to Step 1 until a termination condition is met (no tool call, max rounds, token budget exhausted, guardrail triggered, user interrupt, or safety refusal).

Complex tasks may span many rounds; Anthropic’s two‑stage "Ralph" loop initializes the agent with scripts, progress files, and a git commit, then iteratively encodes the agent, reads git logs, selects the highest‑priority unfinished feature, develops, commits, and writes a summary, using the file system to maintain continuity across context windows.

Real‑World Framework Implementations

Anthropic’s Claude Agent SDK exposes a single query() function that creates the harness, runs the loop, and returns an async iterator of streamed messages. The runtime is a "dumb loop"; intelligence resides in the model. Claude Code follows a "collect‑act‑validate" cycle: collect context (search files, read code), act (edit files, run commands), validate (run tests, check output), repeat.

OpenAI’s Agents SDK provides a Runner class supporting async, sync, and streaming modes. The SDK follows a "code‑first" philosophy: workflow logic is expressed in native Python rather than a graph DSL. Codex Harness builds three layers: Codex Core (agent code + runtime), App Server (bidirectional JSON‑RPC API), and client interfaces (CLI, VS Code, web app). All interfaces share the same harness, explaining why Codex performs better in its own UI than in generic chat windows.

LangGraph models the harness as an explicit state graph with two nodes ( llm_call and tool_node) connected by a conditional edge: if a tool call exists, route to tool_node; otherwise, route to END. LangGraph evolved from LangChain’s AgentExecutor, which was deprecated in v0.2 due to scalability issues. LangChain’s Deep Agents explicitly use the term "Agent Harness" and include built‑in tools, planning ( write_todos), file‑system context management, sub‑agent spawning, and persistent memory.

CrewAI implements a role‑based multi‑agent architecture: Agent (harness around LLM with role, goal, background, tools), Task (work unit), and Crew (collection of agents). CrewAI’s Flows layer adds deterministic backbones, routing, and validation while the crew handles autonomous collaboration.

AutoGen (evolving into Microsoft Agent Framework) pioneered dialogue‑driven orchestration with three layers (Core, AgentChat, Extensions) and five orchestration modes: sequential, concurrent (fan‑out/fan‑in), group chat, handoff, and Magentic (dynamic task board management).

Scaffold Analogy

Just as scaffolding in construction provides temporary access to otherwise unreachable areas without building the structure itself, an Agent Harness enables LLMs to perform complex tasks. When the building (model) is complete, the scaffolding is removed. As models improve, harness complexity should decrease; repeated refactoring shows tool definitions collapsing into generic shells and "agent management" becoming simple handoffs.

Seven Design Decisions for Harnesses

Single vs. Multi‑Agent : Start with a single agent; split only when tool overlap exceeds ~10 or distinct task domains emerge, due to added LLM calls and context loss.

ReAct vs. Plan‑then‑Execute : ReAct interleaves reasoning and action (flexible but higher per‑step cost). Plan‑then‑Execute separates planning from execution; LLMCompiler reports it is 3.6× faster than sequential ReAct.

Context Window Management : Five production strategies—time‑based clearing, conversation summarization, observation masking, structured note‑taking, and sub‑agent delegation. ACON research shows prioritizing reasoning traces over raw tool output retains >95 % accuracy while cutting token usage by 26‑54 %.

Validation Loop Design : Computational validation (tests, code checkers) offers deterministic benchmarks; LLM‑as‑judge provides semantic checks but adds latency. Martin Fowler’s Thoughtworks framework distinguishes "guidance" (pre‑action) from "sensors" (post‑action).

Permission & Security Model : Loose (fast, risky, auto‑approve) vs. restrictive (slow, safe, requires approval). Choice depends on deployment scenario.

Tool Scope Strategy : More tools often degrade performance. Vercel removed 80 % of tools in v0, improving results. Claude Code uses lazy loading to achieve 95 % context reduction. Principle: expose only the minimal tool set needed for the current step.

Harness Thickness : Determines how much logic resides in the harness versus the model. Anthropic bets on thin harnesses and model improvements; graph‑based frameworks favor explicit control. As newer model versions internalize capabilities, Claude Code regularly removes planning steps from its harness.

Harness as Product Differentiator

Two products using the same model can exhibit vastly different performance solely due to harness design. TerminalBench shows that changing only the harness can lift an agent’s ranking by over 20 positions.

Harness is not a solved problem nor a plug‑and‑play component; it is the hard‑core engineering that treats context as a scarce resource, designs validation loops to catch errors before they compound, builds memory systems that avoid hallucination, and makes architectural trade‑offs between scaffolding and model capability.

As model abilities grow, the field moves toward thinner harnesses, but harnesses will remain essential: even the most capable models need infrastructure to manage context windows, execute tool calls, persist state, and verify work.

If your agent fails, don’t blame the model first—examine the harness.