What Is a Harness? Why the Same Model Behaves Differently Across Coding Agents

The piece opens with a community observation: Theo found GPT‑5.6 Sol noticeably more effective inside Claude Code than inside Codex, and Tibo packaged the connection via CLIProxyAPI (CPA) under the alias claudex. The author uses this as a springboard to ask: if the model is unchanged, why does the agent’s behavior and output change?

Connecting the Same Model ≠ Getting the Same Agent

CLIProxyAPI performs protocol translation and request forwarding so Claude Code can call GPT‑5.6 Sol, yet Claude Code does not become Codex. The model still receives system instructions, sees project materials, uses tool descriptions, dispatches sub‑tasks, handles command returns, and compresses history exactly as Claude Code dictates. The claudex alias also sets sub‑agent model, reasoning toggle, and tool concurrency — parameters that belong to the Harness, not the model. Theo’s perceived difference could stem from any of these: system prompt and context shaping task understanding; tool definitions steering action selection; loop and feedback mechanics governing error recovery; concurrency, compression, and sub‑agent strategy limiting how far a task can progress. Without a controlled experiment, the delta cannot be pinned to a single knob.

An official OpenAI ARC‑AGI‑3 experiment provides a reference point: keeping GPT‑5.6 Sol’s reasoning traces and adjusting context compression lifted scores from 13.3% to 38.3% while cutting output tokens to one‑sixth. The numbers don’t transfer directly to coding tasks, but they demonstrate that identical weights under different runtime regimes can yield drastically different results. A fair Harness comparison therefore requires fixing task, model, and acceptance criteria, varying only the outer execution chain.

What Is a Harness?

Earendil’s “What is a Harness?” defines it as the software that supplies an AI model with a runtime environment. The article breaks this into four foundational pieces:

System Prompt — rules and background for the current job.

Tools — search, read/write files, execute code, call external services.

Agentic Loop — chains model calls and tool results so the task advances round by round.

Translation Layer — bridges differences in model interfaces, message formats, and tool‑call schemas.

Once an agent enters a codebase or business system, the question shifts from “can the model call tools?” to “who authorizes execution, what happened, and how do we continue after failure?” The author adds an architectural view: Harness is both the model’s runtime and the execution boundary between model and real systems. The model proposes the next step; Harness turns that step into a constrained, recorded, verifiable system action.

Model Judges, Runtime Executes

In a cross‑module refactor the model decides which files to read, what code to change, and which tests to run. At execution time Harness must handle working directory, file permissions, command timeouts, network access, human approval, and process cancellation. A model’s tool call is merely an action proposal; only after parameter validation and permission checks does it become a real system call. This separation is practical: system prompts can warn against touching production data but cannot replace permission controls; tool schemas constrain format but not business validity; user approval of one command does not isolate child processes. Pi’s security docs explicitly note that Project Trust governs config/extension loading and is not a sandbox — Pi and its TypeScript Extension inherit the launching user’s privileges. For untrusted repos or unattended tasks, true isolation still requires containers, VMs, micro‑VMs, or remote sandboxes. The same logic applies to any Harness: prompts are the first constraint, never the final safety boundary.

The Loop Is Easy; Feedback Is Hard

Agent Loop code is often short: model proposes action → system executes tool → result returns to context → model decides next step. The difficulty lies in what format the result takes, who judges its usefulness, and where to resume after failure. A tool returning only “success” versus one returning exit code, stdout, stderr, duration, and artifact location gives the model completely different decision‑making power. In the refactor example, a test exit code of 0 only means the command finished normally, not that the refactor meets requirements. Harness can persist test results and code diffs, but true completion criteria come from CI, acceptance rules, or code review. This blurs Harness and Environment: Harness orchestrates task execution; Environment is the real world holding code, CI state, tickets, production data, and the feedback after each action. If feedback is unreliable, a faster loop merely chases noise faster.

Context Can Be Compressed; Executed Events Cannot Be Rewritten

Long‑running tasks force Harness to answer: how does the model remember what happened? The naive approach — stuffing all chat and tool output back into context — hits length limits. Old content gets compressed, long logs trimmed, branch materials reordered. The model sees a curated input, not a full execution ledger. Business systems don’t treat UI summaries as databases; agents shouldn’t treat context as the sole source of truth. The article cites concrete implementations: Codex uses Thread / Turn / Item to separate session, task advance, and discrete events; DSH makes Session an append‑only event stream and projects model‑bound messages from it; Pi uses a tree‑structured JSONL Session supporting resume and branching. Implementations differ, but they solve the same problem: context may be compressed, reordered, and rebuilt, but executed commands, tool results, and human approvals must have stable records.

This boundary directly affects recovery. If a write has already reached an external system but Harness crashes before receiving the response, a naive resume that re‑executes because context lacks the result duplicates side effects. Refunds, notifications, ticket creation need business idempotency keys; when results are unknown, Harness must query the authoritative system before deciding retry or manual intervention. Harness need not own business transactions, but it must distinguish “preparing to execute,” “already sent,” “result unknown,” and “confirmed.” Otherwise “resume from breakpoint” just re‑runs side effects.

Translation Layer Solves Connectivity, Not Equivalence

Returning to claudex, CLIProxyAPI’s value is the Translation Layer — different providers use different request, message, and tool‑call formats, so a translation layer is necessary. Yet interface connectivity ≠ behavioral equivalence. Models handle system messages, tool definitions, reasoning traces, prompt caching, and streaming events differently; some fields must be echoed back verbatim in the next round. Pi author Mario Zechner calls cross‑provider context handoff best effort — convert as well as possible, but no semantic‑lossless guarantee. Armin Ronacher documented a similar pivot: a unified SDK proved unable to sustainably flatten model, provider‑side tool, cache control, and history differences, so they switched to directly owning each vendor’s SDK and the Agent Loop. For a Harness, model migration has at least three gates: API call works; task state and artifacts port over; quality, cost, and safety still pass after the swap. The first two can be solved with adaptation and owned data boundaries; the last demands re‑evaluation. This explains Theo’s comparison: it didn’t prove one coding agent universally stronger, but reminded us that plugging the same model into another Harness only guarantees “it runs”; whether it behaves the same way depends on how the entire runtime cooperates.

Architecture Design: Draw Responsibility Lines First

Pi, Codex, and DSH are all Harnesses, yet their internal trade‑offs differ sharply. Pi keeps a thin default core, pushing capability into Extensions, Skills, and user workflows. Codex bundles Thread, Turn, Item, command execution, sandbox, and approval into one execution semantics, exposed via CLI, SDK, and App Server. DSH uses Cordis for a replaceable runtime graph and an event‑stream Session for immutable history. No single design is universally superior: a solo developer watching a short task benefits from transparency; a multi‑hour, multi‑client task touching production systems demands thick state, permissions, recovery, and audit. The author’s design approach is to clarify responsibilities before copying modules:

Who owns business facts vs. who only stores agent work logs?

Who approves actions vs. who performs final permission checks?

After interruption, where to resume and who confirms ambiguous results?

When the model says “done,” which evidence counts as true completion?

After provider, tool, or prompt changes, what re‑evaluates quality?

With boundaries set, module choices become clearer: short tasks may need only a tiny loop plus restricted tools; long tasks require durable state, cancellation propagation, and recovery protocols; business side‑effects demand idempotency, compensation, and audit in the right system.

Back to claudex

Tibo connected GPT‑5.6 Sol to Claude Code in a few config lines. The model can be swapped, the API translated, but the agent’s ultimate behavior is still co‑determined by model and Harness. Harness is the execution system outside the model. The model decides what it wants to do next; Harness decides whether that step can run, how it runs, and what evidence remains after it runs. Models will keep evolving. If task records, tool contracts, run artifacts, and evaluation samples stay in our own system, we can at least explain how a task was completed when the underlying model changes, and we have grounds to re‑choose.

References

Theo: personal experience of GPT‑5.6 Sol in Claude Code vs. Codex

Theo: connecting Claude Code and Codex auth via CLIProxyAPI

Tibo: three‑step claudex recipe

CLIProxyAPI

Earendil: “What is a Harness?”

OpenAI: “Codex as a platform: build on the open agent harness”

Pi official docs: Security, Sessions

Mario Zechner: “What I learned building an opinionated and minimal coding agent”

Armin Ronacher: “Agent Design Is Still Hard”

Anthropic: “Effective harnesses for long‑running agents”