How Meta‑Harness Revolutionizes LLM Harness Optimization with 10× Search Speed

Meta‑Harness, a recent Stanford paper highlighted by industry experts, proposes an external‑loop optimization framework that enables coding agents to automatically search and refine the "harness"—the code that stores, retrieves, and presents information for large language models (LLMs). By granting agents full file‑system access to complete historical experience (source code, execution traces, and scores), the system dramatically improves search efficiency (up to 10×) and overall performance.

Why Optimize the Harness?

LLM performance depends not only on model weights but also on the surrounding harness, which determines what information is stored, when it is retrieved, and how context is presented. Optimizing these decisions can yield up to a six‑fold performance gap, yet current harness engineering relies heavily on manual trial‑and‑error.

Limitations of Existing Text Optimizers

Over‑compress feedback by using only scalar scores.

Access only the current candidate without memory of past attempts.

Restrict feedback to short templates or LLM‑generated summaries.

These constraints are especially harmful because harness decisions exhibit long‑range dependencies; early storage or retrieval choices can affect outcomes many steps later.

Core Method of Meta‑Harness

The key innovation is exposing the full historical experience via a file system, allowing a coding agent (instead of a fixed optimizer) to diagnose and improve the harness.

Search Loop

The loop works as follows:

Agent reads a file system containing all previous harness source code, execution traces, and scores.

Agent proposes a new harness and evaluates it.

All logs are stored in a new directory for future reference.

Key Design Elements

Agentic Proposer : Uses a coding agent (e.g., Claude Code) that can invoke tools like grep and cat to query the file system.

Complete Experience Storage : Each candidate harness directory contains full source code, evaluation scores, and execution traces (prompts, tool calls, model outputs, state updates).

Selective Diagnosis : In each round the agent reads a median of 82 files (≈41% source code, 40% traces) instead of loading everything at once.

Why It Works in Code Space

Structural Impact : Small changes to retrieval or memory logic can have large downstream effects.

Interpretability : By inspecting execution traces, the agent can infer failure causes (e.g., a retrieval step at step 15 causing state pollution).

Natural Regularization : Code models tend to propose coherent algorithms rather than brittle hard‑coded solutions.

Experimental Results

1. Online Text Classification

Evaluated on LawBench, Symptom2Disease, and USPTO using GPT‑OSS‑120B as the classifier.

Accuracy : 48.6 % vs. ACE 40.9 % (↑7.7 points).

Context Efficiency : 11.4 K tokens vs. ACE 50.8 K (4× reduction).

Speed : Achieves ACE‑level accuracy with only 4 evaluations instead of 40 (10× efficiency).

2. Retrieval‑Augmented Math

Tested on 200 IMO‑level problems with a 500 k+ solution corpus.

Single discovered harness improves across five held‑out models (GPT‑5.4‑nano, GPT‑5.4‑mini, Gemini‑3.1‑Flash‑Lite, Gemini‑3‑Flash, GPT‑OSS‑20B) consistently.

Average gain of 4.7 points, surpassing BM25 (+3.4) and Dense Retrieval (+0.3).

3. Agentic Coding (TerminalBench‑2)

Evaluated on 89 high‑difficulty terminal tasks requiring long‑range autonomous execution.

Claude Opus 4.6 : 76.4 % success (2nd place, behind ForgeCode 81.8 %).

Claude Haiku 4.5 : 37.6 % success (1st place, beating Goose 35.5 %).

Key mechanism: Environment Bootstrapping —before the agent loop, a shell command snapshots the OS, installed languages, package managers, and /app directory, injecting this snapshot into the initial prompt and saving 3–5 exploration steps.

In‑Depth Analysis

Information‑Access Ablation

Three access modes were compared:

Score‑only access: 41.3 % best accuracy.

Score + summary: 38.7 %.

Full access (including execution traces): 56.7 %.

Conclusion: Direct access to raw execution traces is the critical ingredient; summarization can discard useful diagnostic information.

Qualitative Study: How Agents Learn

Log analysis from TerminalBench‑2 shows agents performing causal reasoning:

Rounds 1‑2: Simultaneous structural fixes and prompt changes cause regression.

Round 3: Agent isolates the root cause to prompt modification.

Round 7: Switches to pure additive changes (environment snapshot) yielding the best candidate.

Round 8: Combines environment snapshot with early fixes for further improvement.

This ability to identify confounding factors stems from full file‑system visibility.

Sample Harnesses

Draft‑Verification Classification Harness

# Two‑stage process
Stage 1: Retrieve 5 similar examples → generate Draft label D
Stage 2: Retrieve 5 confirmers (D) + 5 challengers (≠D) → verify or correct D

Label‑Primed Query Harness

Label Primer : List all valid labels.

Coverage Block : Most relevant example for each label.

Contrastive Block : Pairs of similar examples with different labels.

https://arxiv.org/pdf/2603.28052
Project page https://yoonholee.com/meta-harness/
Optimized harness: https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact
Meta‑Harness: End‑to‑End Optimization of Model Harnesses