YC Debunks 'Model-Only' Myth: Harness, Not Model, Sets Agent Ceiling

YC's Paper Club session "Why The Harness Matters More Than The Model" challenges the prevailing "model-centric" view by demonstrating that the scaffolding layer—Harness—is the true bottleneck for agent capabilities. The same Claude Opus weights achieve only 30% on ARC-AGI-3 when run bare, but climb to 95% with a well-designed Harness, while NVIDIA's AVO reaches 100%. The model weights remain unchanged; the 65-percentage-point gap comes entirely from the runtime layer.

What Is Harness?

Prime Intellect researcher Seth defines Harness as the layer between the model and the real world that provides persistent state, tools, and compute . The model acts as a sequence processor—predicting the next token—while Harness grounds that reasoning: reading/writing files, calling APIs, managing memory, coordinating sub-agents, and deciding when to continue or stop. Identical models paired with different Harnesses exhibit drastically different behaviors.

Six-Year Evolution: From Bare Loops to Self-Modifying Systems

The presenter condensed key milestones into a capability-oriented timeline:

GPT-2 V0 era (2019): Only a system prompt, context window, and a stop-at-sentence-end loop—no tools, skills, or memory management.

Few-shot & Chain-of-Thought (~2020): Innovations in context space (examples) and output space (explicit reasoning tokens).

WebGPT & Toolformer: First external tool calls—e.g., invoking a subtraction function instead of computing in weights.

MemGPT: Gave the model CRUD access to its own context, enabling long-term memory.

Voyager (Minecraft): Distilled verified tool chains into a skills.md file for retrieval and reuse—the prototype of today's "Skill" concept.

ReAct → Self-Refine → Reflexion: Multi-agent self-critique and environment-feedback loops.

RLM (Recursive Language Model): Main agent recursively spawns sub-agents that can further spawn their own—a key step toward mature "Harness V1".

All above are classified as Harness V1: Static Era —the Harness code itself never changes during execution; only stacked capabilities grow.

Inflection Point: Harness That Learns to Evolve

In the last six months, Harness began to self-improve. Three representative approaches:

DSPy (Demonstrate-Search-Predict): Uses a genetic-programming-style loop to generate, merge, and evaluate candidate system prompts—effectively CRUD on prompts without backpropagation.

Darwin Machine (Darwin Gödel Machine): Goes beyond prompts; mutates the Harness code itself. Maintains an "agent archive" of (Harness code + system prompt) pairs, evaluates via fitness functions, selects, mutates, and reinserts. The meta-Harness produces better Harnesses.

Continual Harness: Refines memory categories (history, skills, memory, sub-agent specs) and adopts Dagger-style online learning—updating Harness config and even test-time training on weights with few shots. Called "an important research direction worth investing in."

These three lines form the technical substrate for the three showcased projects: Prime Agent, Open Jarvis, and QM.

Case 1: Prime Agent — Managing Memory Like CPU Caches

Prime Agent, a self-evolving RLM Harness, structures memory in three tiers analogous to CPU L1/L2/L3 caches:

L1 (Active Context): Immediate working memory.

L2 (Ripple/RAM): Compressed, refined intermediate memory.

L3 (Persistent Storage): Long-term durable memory.

Sub-agents are persistent sub-sessions: they idle in memory after task completion and can be messaged directly without context rebuild, solving the common context-loss problem in multi-agent collaboration.

ARC-AGI-3 tuning journey: First run with a community system prompt hit 99.9%—later revealed as sandbox isolation failure. After fixing sandbox, GPT-5 series scored 78%; other models 70–80%. The team stresses that many benchmarks compare models under unequal budgets (time, tokens), masking true differences; they prioritize "where the long-horizon performance inflection point lies" over single-point scores.

Extreme long-horizon experiment: 8 H200 nodes, 7 days, 633 agents, 23M tokens on an automated factory-build task. Agents never stagnated, continuously advancing the tech tree—dubbed "our own Gemini Plays Pokémon ".

Seth's takeaways: focus on agentic context management (hierarchical memory, compression, refinement), swarm collaboration & deep RLM usage , and standardized long-horizon evaluation instead of single-point metrics.

Case 2: Open Jarvis — Local Stack, 800× Cost Reduction

Stanford's Open Jarvis addresses reliance on cloud LLMs: high API costs, privacy risks, "renting not owning" intelligence. They decompose any personal AI Harness into five primitives:

User Interface

Agent Logic (reasoning + tool use composition)

Intelligence (local model: Qwen, GPT-OSS, Gemma 3N, etc.)

Inference Engine & Hardware (Ollama, llama.cpp, vLLM, SGLang on Apple Silicon or NVIDIA)

Tools, Memory & Learning Primitives (MCP protocol, prompt optimization via DSPy, weight optimization via LoRA/SFT)

Clever design: Use a cloud LLM to automatically optimize the local Harness config—diagnose issues, propose improvements, generate better configs—while inference incurs zero cloud calls.

Results: Optimized local configs significantly beat out-of-the-box local deployments, matching cloud models on personal, coding, and agent tasks; remaining gaps shrink monthly. Key metric: up to ~800× cost reduction with lower latency. Benefits hold across cloud optimizers (Opus, GPT, Gemini, Kimi, GLM), suggesting the "cloud-intelligence-optimizes-local-Harness" path has generality.

Case 3: QM — YC's Internal Four-Generation Harness Evolution

QM is YC's open-source Agent Harness, aiming to give every employee a deeply customizable assistant invokable in Slack or web. Four generations:

Critical pivot after "Hermes Fleet": Binding agent "brains" to specific sandbox VMs seemed powerful but proved fragile—management overhead exploded with sandbox count, and session records were locked inside individual machines.

Solution: Offload all conversations and state to Postgres , expose to agents for direct access. Sandboxes become on-demand resources—heavy dev environment → powerful sandbox; simple task → lightweight sandbox—agent decides, not hard-coded Harness logic.

Pragmatic engineering lessons:

Agents quit too early: Introduced a "grind tool" enforcing minimum time/token budget before allowing task abandonment—significantly improved long-report quality. OpenAI and Anthropic reportedly use similar tricks for math breakthroughs.

Agents lose situational awareness: Even with clear system prompts, agents confuse context in multi-user Slack threads; need extra "local awareness anchors" for correction.

Missing permissions & social norms: Humans intuitively know what info to share with whom; agents leak private data. QM leverages YC's fine-grained permission system, but most companies lack such infrastructure—significant engineering investment needed.

Human review becoming rubber-stamp: Early DB write proposals were carefully reviewed; as trust grew, reviews turned perfunctory—flagged as a risk for coming months, not a comfort signal.

Cross-Case Design Principles

Push decisions to the agent, not hard-code in Harness: Let agent choose sandbox, model provider, etc.—outperforms Harness-level mandates.

Layered memory management, not infinite context stacking: Active context → Ripple/RAM → persistent storage; compression and refinement as continuous processes.

Keep Harness minimal, retain only core capabilities: QM converged on three core tools: remote sandbox execution, object storage read/write, internal app publishing; everything else treated as transitional patches.

Give agents endurance—don't let them quit at first setback: Budget-constrained "grind" mechanism is a simple but effective lever for long-horizon task quality.

Trust must be paired with continuous audit mechanisms: Otherwise human-in-the-loop degrades into theater.

Closing Thought

The core message is straightforward: model IQ curves have risen steadily, but converting that IQ into usable productivity depends on the long-underestimated engineering layer—Harness. The leap from 30% to 95% to 100% on ARC-AGI came not from smarter models but from giving the same model a better "body" that knows how to use it. As marginal model gains slow, Harness is emerging as the next true competitive high ground—these three projects make that answer abundantly clear.