Turning One‑Shot AI Agents into Evolvable Systems with Harness Engineering

Why Prompt Engineering Is Not Enough

Prompt engineering only tells the model what to do for a single turn (e.g., "output in Chinese", "list risks then solutions"). When an AI agent calls tools, modifies files, or performs multi‑step workflows, the quality of the result depends on the entire runtime environment , not on a longer prompt.

Harness Engineering

Harness Engineering is the practice of designing, implementing, and continuously improving the execution environment for AI agents. It bundles task specifications, context, tools, permissions, memory, skills, workflow, verification, logging, and feedback so that an agent becomes a stable, replayable, and evolvable system rather than a one‑off conversation.

Eight Core Actions of a Harness

Specify : turn the goal into an executable specification (goal, scope, non‑goal, constraints, acceptance criteria, required outputs).

Ground : provide the agent with a trustworthy context package (task description, required files, trusted fact sources, recent changes, known risks).

Equip : list the tools the agent may use, each with a clear input schema, JSON/table output, explicit error codes, low‑risk default permissions, and observable metrics.

Constrain : define permission levels (read‑only, low‑risk write, medium‑risk change, high‑risk action) and enforce approval gates for risky operations.

Orchestrate : arrange execution order, hand‑off contracts, and stage boundaries.

Verify : deterministic checks (tests, lint, schema validation) plus quality judgement (content checks, human review for high‑risk decisions).

Observe : record tool calls, parameters, latency, runtime logs, and produced artifacts for post‑mortem analysis.

Improve : turn verified experience into rules, tests, skills, memory entries, or tool upgrades.

Controlled Self‑Evolution

Self‑evolution is not arbitrary model “learning”. A safe engineering loop consists of:

Traceable experience capture (input, output, tool calls, failures, verification results, user corrections).

Failure attribution to a specific layer.

Generation of concrete improvement candidates.

Gate‑controlled validation (golden tasks, human review, rollback).

Promotion of verified changes or safe rollback.

Nine‑Layer Harness Structure

Each layer must answer three questions: what signal indicates a change, how should it change, and how to validate the change.

1. Task Specification

A good spec defines:

Goal : e.g., reduce search‑service P95 latency.

Scope : only modify the search service and related tests.

Non‑goal : do not change DB schema or API shape.

Acceptance : existing tests pass and benchmark shows ≥30% latency reduction.

Output : description of changes, risks, verification command and result.

2. Context

Context should be concise and trustworthy. Example package:

# Context package
## Task
Fix duplicate payment callback issue.
## Must‑read files
- payment/callback.go
- payment/idempotency.go
- docs/payment-design.md
## Trusted sources
- DB schema from migrations/
- API behavior from openapi.yaml
- Recent incidents from incident docs
## Known risks
- Do not modify accounting logic
- Do not bypass idempotency key

3. Tools

Good tools have:

Input : explicit parameters with a schema.

Output : JSON, tables, or structured logs.

Error handling : clear error codes and failure stage.

Permission model : low‑risk by default, high‑risk requires approval.

Observability : record latency and a result summary.

4. Permissions

Read‑only : allowed by default (read files, query status, search history).

Low‑risk write : create drafts or temporary files, auto‑execute, record path.

Medium‑risk change : modify code or create remote docs, must verify after execution.

High‑risk action : delete, overwrite, publish, or production changes; require human approval and rollback plan.

5. Memory

Memory stores stable preferences, long‑term rules, environment paths, and verified failure experience. It must NOT store temporary facts, unverified guesses, clear‑text credentials, or expired links.

6. Skills

When an agent repeats a process, distill it into a Skill that lists when to use it, inputs, steps, tools, outputs, verification, safety boundaries, and common failures.

7. Workflow

Long tasks are split into stages with explicit hand‑off contracts. Example hand‑off package:

# Handoff package
## Current goal
Complete search‑service performance optimisation.
## Completed
- Identified slow query in user_profile join
- Added benchmark
## Pending
- Add boundary tests
- Run full‑scale test
- Write risk analysis
## Verification commands
- go test ./search/...
- go test ./...

8. Verification

Two‑step verification:

Deterministic checks : tests, lint, type‑check, schema validation.

Quality judgement : content checks (word count, field completeness, link reachability), semantic checks (answers the question, provides evidence), and human review for high‑risk decisions.

9. Observability

Observability records what the agent did:

Trace : tool call, arguments, return value, latency.

Runtime log : what happened, what was encountered, how it was solved.

Artifact : report, CSV, code diff, screenshot, evidence ledger.

Scenario‑Specific Harnesses

Coding Agent Harness

Focus on scope, tests, and diff review. Minimal workflow: understand requirement → fetch relevant files → write plan → incremental changes → run tests → diff review → output risk and verification.

Doc/README → markdown lint or manual check.

Single‑function bugfix → relevant unit test.

Shared library change → module test or full test suite.

API behaviour change → unit + integration test.

Performance optimisation → benchmark or profiling.

UI change → browser open, screenshot, or interaction check.

Research Agent Harness

Emphasise source provenance, evidence tracking, and clear separation of fact, inference, judgement, and recommendation. Every conclusion must be traceable to an official document, paper, standard, or raw data.

Data Agent Harness

Key items to record:

Source : DB, API, file, time window, query parameters.

Raw data : file, record count, fields, checksum.

Cleaning process : filter conditions, field mapping, outlier handling.

Statistical definition : denominator, numerator, dedup rules, time bounds.

Output : chart, table, conclusion, unverified items.

Long‑Running Agent Harness

Guarantee no state loss by defining stages, status checkpoints, recovery points, and a classified failure list (permission, data, conflict, tool). Completed work is never repeated; failures are categorized for targeted remediation.

Building a Minimal Harness

Start with a concrete, high‑frequency task that has a clear verification method. Follow these steps:

Select a high‑frequency task.

Write a detailed task specification.

List required context files.

Enumerate the tool inventory.

Define permission boundaries.

Design the workflow (stages and hand‑off).

Design verification (deterministic + semantic checks).

Record runtime logs and artifacts.

Perform a failure post‑mortem.

Codify the experience into a rule, skill, memory entry, eval, or tool improvement.

The minimal loop is input → execute → verify → record → improve. Iterate from a coarse v0 to scripted steps (v1), added eval (v2), and finally long‑term memory and multi‑agent coordination (v3).

Maturity Model

Level 0 – Model can answer : uncontrolled, suitable only for brainstorming or one‑off drafts.

Level 1 – Agent can act : unstable automation, suitable for small‑scale scripts or simple code changes.

Level 2 – Rules & verification : stable but no self‑learning; suitable for daily engineering collaboration.

Level 3 – Memory & skills : still lacks sufficient metrics; suitable for repetitive tasks and complex research.

Level 4 – Eval & closed loop : production‑grade agents with multi‑person collaboration; rollout cost is a concern.

Level 5 – Controlled self‑evolution : platform‑level intelligent system with strong evaluation; governance complexity is the main hurdle.

Seven‑Day Practice Roadmap

Explain the Harness concept and identify failure layers.

Write a task spec and context package (goal, scope, non‑goal, acceptance, trusted sources).

Design tool list and permission levels (low‑risk auto‑exec, medium‑risk requires approval, high‑risk gated).

Design verification (deterministic checks, semantic checks) and logging strategy.

Draft a Skill and a reusable Memory entry.

Build the minimal Harness and run a real task.

Post‑mortem the run and generate an improvement candidate.

Key Risks and Mitigations

Over‑fitting : only a few examples improve; mitigate by keeping diverse golden tasks.

Self‑pollution : erroneous conclusions written to memory/skill; mitigate by keeping unverified conclusions in a candidate pool only.

Rule explosion : every failure adds a rule, bloating the system; mitigate by merging similar rules and periodic pruning.

Automation over‑privilege : efficiency pushes high‑risk permissions; mitigate by requiring human approval for high‑risk actions.

Metric cheating : eval scores improve while real performance degrades; mitigate with human spot‑checks and real‑task replay.

Context collapse : repeated summarisation loses details; mitigate by preserving evidence links and raw examples.

Conclusion

Harness Engineering is not about locking an agent down or achieving full automation. It is about ensuring that an agent can reliably deliver on real tasks, that failures are visible, that validated experience is fed back into the system, and that the system improves without contaminating its own knowledge base. Moving from prompt engineering to harness engineering means building a runnable system with a closed feedback loop: input → execute → verify → record → improve.