How Meta-Harness Enables AI to Self‑Optimize Its Own Harness

Engineers often spend more time tuning prompts and designing workflows for AI than the AI actually executes. The Stanford IRIS Lab team introduces Meta‑Harness, an open‑source framework that lets the AI act as an intern with unrestricted access to proprietary information.

Core Mechanism: the “time machine”

Meta‑Harness records three artifacts for every attempt:

Source code : the complete tool‑chain code used in the attempt.

Execution trace : detailed logs and intermediate results produced during code execution.

Evaluation score : the final performance on a standard test set.

Providing the AI with the full historical record enables it to trace long‑term dependencies that may span dozens of prior attempts.

Empirical evidence

Experiments on the TerminalBench‑2 benchmark compared full‑history access with summary‑only methods such as TTT‑Discover and Best‑of‑N. The summary‑only approaches performed significantly worse. On average the AI needed to read 82 historical files to locate the true failure mode and generate targeted hypotheses.

Performance results

On a text‑classification task, Meta‑Harness reached the final performance level of competing methods after only four evaluation rounds . On TerminalBench‑2 it achieved approximately 55 % accuracy, surpassing reported Claude Haiku 4.5 suites that ranged between 40 %–50 % .

Technical challenges

Time cost : each evaluation round requires 3 hours even with maximal parallelism.

Data explosion : a single evaluation produces roughly 10 million tokens of raw output, far exceeding any model’s context window.

Storage design : a dedicated file system is needed to manage the continuously growing “experience repository.”

Analysis of model vs. toolchain

The team notes that LLM systems consist of model weights and the surrounding toolchain. For hard problems, the toolchain’s quality can dominate performance, as evidenced by the extensive prompt‑engineering and agent‑design effort already invested.

Generalization and over‑fitting concerns

Memory‑optimizing the objective in text space is inefficient compared to weight‑space encoding. The authors validated generalization by retaining task instances, testing on unseen backbone LLMs, and evaluating on new domains.

Conclusion

Meta‑Harness itself is a harness that specializes in optimizing other harnesses, embodying a meta‑learning paradigm that dramatically reduces manual structural constraints, allows the AI to review its full historical experience, and autonomously explores optimization directions.

Paper: https://arxiv.org/pdf/2603.28052

GitHub repository: https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact