Why Changing an AI Harness Can Make a Model Appear Dumber—and How EverMind Makes Agents Smarter

Even when a model’s weights remain unchanged, swapping its surrounding harness—prompt, knowledge injection, tool calls, runtime logic, and configuration—can cause large performance swings, exposing two key questions for self‑evolution: how to prove that improvements are genuine and how to obtain continuous, real, traceable data for long‑term evolution.

EverMind addresses these questions with a product pathway that links HarnessBank, Raven, and EverMe, turning academic research into usable AI systems.

What is a harness? In AI agents, the harness is the “runtime shell” that combines prompts, injected knowledge, tool invocation, and configuration with the core model. Different harnesses can make the same model behave like the same person in different organizations: the knowledge stays the same, but decision processes and feedback mechanisms change, leading to divergent outcomes.

In HarnessBank experiments on the AppWorld test set, a frozen Qwen3.6‑27B model improved its Pass@1 score from 41.3 % to 56.7 % solely by evolving the harness. Across seven benchmarks covering terminal operations, code generation, mathematical reasoning, web research, knowledge work, application control, and code repair, Pass@1 rose by 5.1 %–15.4 % .

The real difficulty is not achieving a single score bump but proving that the bump stems from a reusable mechanism rather than random variance, task‑specific over‑fitting, sandbox crashes, or validator timeouts. Without such proof, self‑evolution risks becoming an automatic regression generator.

HarnessBank core architecture splits the system into two asymmetric agents:

Task Agent : Executes tasks under the current harness using a frozen backbone model (default Qwen3.6‑27B), ensuring stability.

Evolver Agent : A stronger model (e.g., Claude Opus 4.8) reads the Task Agent’s execution traces, diagnoses recurring failure mechanisms, and generates new candidate harnesses.

This design lets a “strong model” think about improvements while only the “weak model’s” shell is modified.

Evolution is restricted to the mutable surface of the harness (prompt, knowledge, runtime logic, configuration). Evaluation code, bookkeeping, and self‑evolution logic are locked as an immutable kernel , guaranteeing comparability before and after changes.

The evolution follows a rigorous four‑step loop :

Select parent : Choose the strongest harness from the gene bank using a quality‑biased sampling.

Run diagnostics : Execute a batch of training tasks to collect scores, trajectories, and evaluation metadata.

Generate offspring : The Evolver either invents a new mechanism from failure traces or recombines mechanisms from different gene‑bank cells.

Gated screening : Offspring must pass four strict gates before full evaluation and entry into the bank.

Three common pitfalls of existing self‑evolution approaches and EverMind’s mitigations are:

Search collapse : Greedy evolution narrows changes to conservative prompt tweaks. EverMind uses a MAP‑Elites‑style gene bank that partitions candidates by “what is changed” (prompt, knowledge, runtime, config) and “why it is changed” (failure pathology), preserving diversity.

Task over‑fitting : Without gating, improvements may only memorize training tasks. HarnessBank isolates the test set from the evolution loop, using it solely for final validation.

Unverifiable gains : Noise, sandbox crashes, or validator timeouts can produce false positives. The four‑gate screening—including a significance‑test gate—eliminates phantom progress; ablations show that removing the significance gate yields 62 %–76 % phantom improvements, whereas the full gated version cleanly exits after ten rounds.

Core insight : There is no universal harness. Cross‑model experiments reveal a “pathology‑to‑patch matching law”: each model has dominant failure modes. For example, Qwen3.6‑27B suffers from “empty‑turn rounds” and gains +15.4 % with a verify‑finalize patch, while Qwen3.6‑397B and Gemini 3 Flash are prone to “carelessness” and benefit from checklist patches. Applying a patch designed for one model to another can produce near‑zero improvement or even a –15.7 % regression.

The transferable artifact is the entire pipeline—from diagnosing failures, through structured search, to statistical validation—not any single harness or prompt.

Four practical recommendations for agent developers :

Validate with four questions: Is the evaluation environment clean? Did the change actually execute? Is the gain statistically significant? Is it stronger than the baseline?

Store harnesses in a grid (by change location and failure type) rather than keeping only the current best.

Avoid seeking a universal optimal harness; instead, carry the diagnosis‑to‑validation workflow across models.

Account for cost: strong models and gating incur compute expense, so weigh the risk of deploying a “seeming” improvement against potential regression.

Raven extends HarnessBank’s methodology into the runtime, allowing agents to continuously evolve without frequent weight updates. EverMe, the first C‑side product, provides a Memory Hub, Knowledge Base, Digital Avatar, and Agent Hub, enabling users to collect, view, trace, and reuse personal memories across agents.

For self‑evolution to scale beyond the lab, continuous, real user data is essential. Quantity alone is insufficient; data must belong to a single user, span long periods, include full task processes and feedback, and be manageable by the user. EverMe creates a “personal data flywheel” that differs from traditional internet flywheels by focusing on high‑quality, individualized experience rather than sheer volume.

Thus, EverMind is building a complete path from academic research (HarnessBank), through open‑source infrastructure (Raven), to a consumer‑facing product (EverMe), aiming to deliver AI systems that become smarter the more each individual uses them.