How to Build an AI Agent That Won’t Fall Apart with Harness Engineering

Why Agents Fail and What a Harness Is

Many developers see an agent fail and first try to rewrite the prompt, then change models or enlarge context windows, yet the agent still forgets decisions, picks the wrong tool, or loops endlessly. The root cause is usually not model intelligence but the absence of a reliable surrounding runtime, which the author calls a Harness . Designing and continuously improving this runtime is Harness Engineering .

“Of course, you need an interface and a Harness to use these models.” – Dario Amodei, Anthropic

Models are pure inference engines; a Harness decides what the model can see, which tools it can use, what information persists across sessions, what counts as valid evidence, and when a run should stop.

MODEL (model)
  responsible for reasoning and proposing actions

HARNESS (runtime)
  selects context
  exposes tools
  saves state
  enforces policy
  checks results
  records trace
  recovers from failures

Prompt engineering improves how instructions are written , while Harness Engineering improves under what conditions those instructions are executed .

Seven Core Tasks of a Production‑Grade Harness

1. Turn Requests into Contracts

Before acting, an agent converts a natural‑language request into a bounded task object that defines goal, inputs, output, constraints, and completion criteria.

{
  "goal": "Deliver this feature",
  "inputs": ["issue description", "code repo", "design spec"],
  "output": "reviewable Pull Request",
  "constraints": ["no DB schema changes", "keep public API stable"],
  "done_when": ["tests pass", "visual check passes", "code review passes"]
}

The contract prevents silent task rewrites; without it an agent might claim success while completing the wrong job.

2. Give the Agent a Map

Instead of loading all documentation into the context window, place a concise guide at the repository root that tells the agent where to find details.

AGENTS.md
  ├─ Architecture map
  ├─ Test map
  ├─ Product rules
  ├─ Security rules
  └─ Task‑specific guides

Maps keep context small; detailed knowledge lives close to the code, test, or workflow that needs it.

3. Expose the Right Tools in the Right Environment

Each tool must have a clear purpose, predictable output, explicit failure states, and defined permission boundaries.

Read file          – default allow
Run tests          – sandbox only
Write file         – workspace only
Network access     – limited by task
Deploy             – requires approval
Delete data        – requires approval

Good tools reduce ambiguity before the model starts reasoning; bad tools return vague results that force the model to guess.

4. Externalise Memory as Persistent State

Decisions, artifacts, failures, and unresolved risks should be stored outside the conversation window so that subsequent sessions inherit the true state.

{
  "task_id": "task_042",
  "current_step": "verify_ui",
  "artifacts": ["build.zip", "report.md", "screenshot.png"],
  "decisions": ["keep existing DB schema"],
  "failures": ["overflow at 390px width"],
  "pending": ["await human approval"]
}

This prevents loss of information across context resets, crashes, or hand‑offs.

5. Add Sensors Before Granting Autonomy

Quality signals such as tests, linting, visual checks, source verification, and schema validation turn vague “looks good” into concrete evidence.

code      → tests + type check + lint
UI        → render + screenshot + visual check
research  → source verification + contradiction check
data      → schema + range + freshness check

The Harness evaluates this evidence before allowing the next step.

6. Enforce Policy Outside the Model

The model may suggest an action, but the Harness decides whether to authorize it.

MODEL SUGGESTS  →  POLICY CHECKS  →  TOOL EXECUTES

This separation is crucial for high‑cost, irreversible, or multi‑user actions.

7. Record Trace and Support Partial Recovery

Every run should log request, selected context, tool calls, state changes, verification results, retries, cost, final artifact, and rollback point.

request
selected context
tool invocation
state change
verification result
retry
cost
final artifact
rollback point

Without a trace, failures remain mysteries; with a trace, they become inputs for the next Harness improvement.

Turning Important Instructions into Infrastructure

Instead of relying on a single document that agents may miss, encode critical rules twice: a human‑readable guide and an enforceable check.

GUIDE – "UI layer must not query the database directly."

CHECK – if UI imports Repository, lint fails immediately.

A failed check becomes a permanent system upgrade, so future agents inherit the rule automatically.

Loops Belong to the Harness

Long‑running tasks need bounded loops with evidence, retry limits, budgets, and upgrade paths.

for (let attempt = 1; attempt <= 3; attempt++) {
  const artifact = await build(state);
  const evidence = await verify(artifact);
  if (evidence.pass) return artifact;
  state.failures.push(evidence.gap);
  state.repair = evidence.repair;
}
return requestHumanReview(state);

The Harness decides whether another attempt is allowed.

Failures Should Upgrade the System

Common failure patterns map to systematic Harness improvements:

Missing context          → add map or retrieval rule
Wrong tool               → improve tool description or routing
Unacceptable output     → add validator or tighter contract
Infinite loop            → add retry cap and human escalation
Unsafe action            → add permission gate
Lost decision            → write to persistent state
Unknown failure          → increase tracing and evidence capture

Temporary patches fix a single run; Harness changes improve every future run.

“Good Harnesses turn Agent errors into infrastructure.”

Separate Brain, Hands, and History

When the reasoning model (brain), execution sandbox (hands), and append‑only log (history) are independent, agents become easier to understand, audit, and recover.

BRAIN – inference model
HANDS – sandbox and tools
HISTORY – additive event log

Even if the sandbox crashes or the model is swapped, the other components remain functional.

Managed Agents Architecture

Anthropic’s later design splits responsibilities into three stable interfaces:

Session – stores the append‑only log

Harness – invokes the model and routes tool calls to infrastructure

Sandbox – provides the execution environment for code and file modifications

Each can be replaced independently.

Give Every Run a Change Receipt

A compact receipt records how the final artifact was produced, enabling regression analysis and audit.

{
  "context_sources": ["issue", "repo_map", "design_spec"],
  "policy_version": "v12",
  "model_route": "complex_coding",
  "tools_used": ["shell", "browser", "tests"],
  "tests": {"passed": 42, "failed": 0},
  "human_corrections": 1,
  "retries": 2,
  "cost_usd": 3.84,
  "accepted_artifact": "pr_1842",
  "rollback_point": "commit_7f3a"
}

This lets different model versions be compared and ensures that a seemingly perfect answer does not hide a broken execution path.

Start with the Smallest Closed‑Loop Harness

Build incrementally:

LEVEL 0 – prompt + model
LEVEL 1 – project guide + tools
LEVEL 2 – structured state + tests + bounded loops
LEVEL 3 – policy + trace + recovery + human gate

Only add complexity when the task truly requires it.

Harness Engineering Checklist

Are success criteria defined before execution?

Can the agent locate the right project knowledge without loading everything?

Does each tool have a clear contract and failure mode?

Is the execution environment isolated from production?

Are important decisions stored outside the conversation?

Is evidence captured for every high‑risk state transition?

Are irreversible actions protected by approval?

Does every loop have a retry limit and budget?

Can the run recover after interruption?

Are tool calls and state changes explainable?

Do failures trigger updates to guides, tests, tools, or policies?

Can the final artifact be rolled back?

If many answers are negative, a stronger model alone won’t make the system reliable; the Harness must be improved.

Key Insight

The real competitive edge lies in how quickly a team can encode failures as permanent rules. Over‑engineering with many plugins does not guarantee reliability; a clear contract, persistent state, deterministic checks, and a change receipt are often sufficient.

However, a Harness can also solidify outdated assumptions. As model capabilities evolve, old scaffolding may become a hindrance, so versioning, A/B experiments, and pruning stale rules are essential.

References

Harness engineering: leveraging Codex in an agent‑first world

Harness design for long‑running application development

Scaling Managed Agents: Decoupling the brain from the hands

How Claude Code works

Dario Amodei interview on Claude Code origins