Why the Same Model Feels Different in Coding Agents: Model Sets the Capability Ceiling, Harness Sets the Production Floor

TL;DR

This piece focuses on how to operate a Harness once it starts carrying production load, rather than merely defining what a Harness is.

Agent quality is judged by the "model + Harness" combination, not by model score alone.

Cursor moved from static, heavyweight context to dynamic, on‑demand context.

Evaluation uses both offline CursorBench and online A/B, Keep Rate, and semantic follow‑up signals.

Tool errors are treated as a first‑class reliability problem.

One Harness cannot serve all models; each model requires tailored tool schemas, prompts, and caching policies.

Future multi‑agent systems depend on Harness‑level scheduling, description, and result merging.

For a first‑version Harness, start with ten pragmatic actions (task typing, result retention, combined offline/online evaluation, error classification, context budgeting, versioned model adaptation, safe model switching, sub‑agent routing, dead‑weight cleanup, and failure‑driven design).

Treat Harness as an Online System

Although a Harness is not a traditional API service with databases or queues, the operational challenges are similar. Teams must know when the system slows, why it slows, who is affected, and whether a rollback is possible.

Typical adjustments—prompt tweaks, tool additions, model swaps—can change token usage or cause the system to miss critical context. For example, a prompt change may save tokens but cause long tasks to miss key context.

Cursor treats the Harness as a continuously evolving software product: every change is accompanied by a hypothesis, an offline regression test, online feedback, and a decision to roll back, tune, or discard the patch.

Context Management: From Up‑Front to Dynamic

In early 2024, Cursor’s coding agents required extensive static context: lint results, type errors, directory listings, and compressed user attachments were injected into every request. This made the window noisy and expensive.

Now the static payload is trimmed to low‑cost, high‑value items (OS state, git status, recently viewed files). The Harness fetches additional information on demand via tools, reducing token waste while preserving the ability to retrieve needed data.

The shift embodies the principle that "more context is not always better; better context retrieval is." This aligns with research such as Chroma’s Context Rot and Liu et al.’s Lost in the Middle , which show that long, unstructured windows degrade model performance.

Evaluation Beyond Benchmarks

CursorBench, the internal benchmark suite, constructs realistic tasks from real Cursor sessions, measuring correctness, code quality, efficiency, and interaction behavior. The suite is refreshed continuously to reflect evolving developer usage.

Offline scores are insufficient: an agent may look perfect in a lab but produce code that developers immediately delete. Therefore Cursor adds online experiments that track Keep Rate (how many generated changes remain in the codebase after a fixed period) and Follow‑up Semantic Judgment (whether the user’s next utterance indicates satisfaction).

These signals are portable: a customer‑support agent could use Keep Rate as "repeat‑question rate", a writing agent could use "final paragraph adoption rate", and a data‑analysis agent could use "SQL execution rate".

Tool‑Error Classification as Reliability Engineering

Tool failures are not merely "bad prompts"; they leave noisy tokens that poison subsequent reasoning. Cursor classifies errors into categories such as InvalidArguments, UnexpectedEnvironment, ProviderError, UserAborted, and Timeout. Unknown errors are treated as bugs, while expected errors have per‑tool, per‑model baselines; exceeding a baseline triggers alerts.

Reliability metrics are sliced by tool, model, task type, code language, and repository size, mirroring the front‑end practice of inspecting P99 latency per device.

Model Switching Is Not Just a model id Change

Each model has distinct tool‑format expectations and prompt sensitivities. OpenAI models favor patch‑style edits; Anthropic models prefer string replacement. Consequently, Harness must maintain a versioned configuration for each model, including tool schemas, prompt versions, context budgets, error baselines, and allowed task lists.

When a new model is introduced, Cursor starts from the closest existing Harness configuration, runs offline regression, dog‑foods internally, and iterates on prompts and tool schemas until the "model + Harness" combo passes release criteria.

This co‑evolution is echoed in Anthropic’s "Harnessing Claude’s intelligence" and Manus’s five‑round Harness refactorings.

Multi‑Agent Scheduling and Isolation

Future AI‑assisted software engineering will involve multiple specialized agents (planner, editor, debugger). The system must decide which agent to invoke, how to describe the task for its strengths, and how to merge results back into a coherent workflow. This is fundamentally a Harness responsibility, not a team‑structure issue.

Sub‑agents are useful for isolating high‑output tasks (e.g., log search, documentation lookup) and for providing dedicated tool permissions. They should be described like routing rules, specifying responsibility, trigger conditions, and exclusions.

First‑Version Harness Checklist

Identify task types first; defer role design.

Measure whether results persist (code stays in repo, docs become final, etc.).

Combine offline regression with online feedback; keep a replayable task set.

Classify tool errors into at least eight categories and set per‑tool baselines.

Expose context budget per agent/tool; provide previews for large outputs.

Version each model’s prompt, tool schema, caching, and task suitability.

Handle mid‑task model switches as state migrations, using summaries or fresh sub‑agents.

Write sub‑agent descriptions as explicit routing rules.

After a model upgrade, clean dead weight: remove obsolete rules, compressions, and patches.

Turn every failure into a Harness design entry (e.g., add to AGENTS.md or introduce a hook).

Key Takeaways

Operating an Agent in production is less about chasing the newest model and more about maintaining a robust Harness that provides observable, roll‑backable, and model‑aware engineering loops. Continuous evaluation, fine‑grained error classification, dynamic context, and versioned model‑specific configurations are the core practices that keep AI coding agents reliable and performant.

References

Cursor: "Continually improving our agent harness" https://cursor.com/blog/continually-improving-agent-harness

Cursor: "How we compare model quality in Cursor" https://cursor.com/blog/cursorbench

Anthropic: "Harnessing Claude’s intelligence" https://claude.com/blog/harnessing-claudes-intelligence

Chroma Research: "Context Rot" https://www.trychroma.com/research/context-rot

Liu et al.: "Lost in the Middle" https://arxiv.org/abs/2307.03172

Addy Osmani: "Agentic Engineering" https://addyosmani.com/blog/agentic-engineering/