Why Model Power Isn’t Enough: Inside Anthropic’s Harness for Building Real AI Applications

Model Strength vs. Real‑World Usability

Anthropic’s recent blog demonstrates that even a strong model like Claude can quickly generate a superficially complete game editor, but the resulting application often fails to run correctly, exposing a gap between impressive code generation and functional software.

What Is Harness?

Harness is a scaffolding system surrounding the model, composed of three roles:

Planner : Takes a high‑level instruction (e.g., “build a 2D game editor”) and breaks it into detailed tasks, iterations, and optional AI‑generated features.

Generator : The model itself that implements each task by writing code, assembling UI, and adjusting styles.

Evaluator : An autonomous agent equipped with a Playwright browser that runs the generated app, performs user‑like interactions, captures screenshots, and produces a detailed quality‑assessment report.

The three roles interact in a closed feedback loop, allowing continuous refinement until the application meets predefined quality criteria.

Iterative Development and Cost

In Anthropic’s experiment, a solo Claude run produced a rough editor in 20 minutes for $9, but the app was non‑functional. Using the full Harness, the same goal required six hours and $200, yet yielded a fully playable editor with sprite animation, AI‑generated assets, and export capabilities.

Training a Picky Evaluator

The Evaluator is not a raw Claude instance; it is heavily fine‑tuned with a scoring rubric covering design quality, originality, craftsmanship, and functional completeness. It learns to be critical, pointing out concrete issues such as missing event handlers or default styling.

Dynamic Harness Evolution

As model capabilities grow, Harness adapts. Early versions needed a “context reset” to avoid the model’s “context anxiety” during long tasks. Later models (Opus 4.6) eliminated this need, allowing uninterrupted execution and demonstrating that Harness design space shifts rather than shrinks with model improvements.

Key Engineering Questions

How to decompose vague requirements into stepwise, model‑executable tasks?

How to design an evaluator that provides actionable, picky feedback?

How to balance cost, time, and quality in the iterative loop?

How to continuously refactor the Harness as model abilities evolve?

“As models keep getting stronger, the design space of Harness does not shrink; it moves. AI engineers must keep finding the next point where the model falls short and the Harness can fill the gap.”

The overall insight is that the real bottleneck in AI application development is shifting from raw model capability to the engineering of a robust Harness that can translate that capability into reliable, deliverable software.

https://www.anthropic.com/engineering/harness-design-long-running-apps