Agent Runs Tests ≠ Trustworthy Results: Building Independent Verification in AI-Native Development

The previous article discussed how project context, Skills, Hooks, CI, and platform strategies bring organizational knowledge into Agent execution. These mechanisms let Agents automatically run builds and tests, but a passing test command does not equal reliable functional verification — especially when the same Agent writes code, modifies tests, and summarizes results, it can unintentionally lower the completion standard to match the current implementation.

The key to verification is not who runs the tests, but whether the evidence is independent, objective, and supports the conclusion.

Development Agent Self-Verification Is Important but Insufficient

Using the batch import case: after the development Agent completes upload, parsing, validation, and async tasks, it runs unit tests, integration tests, and build checks. It fixes field mapping errors and adds missing task status implementations and tests. This self-verification is a necessary part of the development loop — without it, many low-cost issues would flow into later reviews. However, self-verification only drives implementation convergence; it cannot alone carry final acceptance of critical functionality.

Verification System Requires Multiple Independent Layers

A more complete verification structure can be divided into five layers:

Development Agent self-verification — quickly finds implementation defects and makes local corrections; supports judgment that the current implementation passes its own executed checks.

Independent verification role (verifier) — checks differences and run results against a predefined baseline; supports judgment whether the implementation meets pre-defined technical conditions.

Test role — designs counter-examples, cross-checks design against implementation; supports judgment whether key scenarios, exceptions, and boundaries are covered.

CI toolchain — repeats deterministic checks in a controlled environment; supports judgment whether a specified version passes fixed gates.

Business acceptance — judges whether the result satisfies real business goals; supports judgment whether the current version can be accepted by the business.

The independent verification role and development Agent use different responsibilities and contexts; the verifier does not participate in generating the current implementation and focuses on finding deviations, omissions, and unverifiable conclusions. The two roles differ in focus: the verifier checks conformance to an agreed baseline, while the test role actively challenges whether the baseline is complete, scenarios are sufficient, and exception boundaries are verifiable.

Role independence does not necessarily require different models, nor does it guarantee zero shared bias. True independence comes from pre-frozen acceptance criteria, controlled toolchains, distinct responsibility boundaries, and evidence sources the implementer cannot arbitrarily modify.

What Different Verification Methods Can Prove

"Tests pass" is too vague. Different checks cover different factual levels:

Unit tests — prove field validation, state transitions, and local rules meet expectations; cannot prove real components collaborate correctly.

Integration tests — prove file storage, task system, and database can cooperate; cannot prove production-scale and long-term stability.

Contract tests — prove front-end/back-end, service, or message input/output compatibility; cannot prove business rules themselves are correct.

UI screenshots and browser checks — prove pages render in specified states and interactions are reachable; cannot prove backend side-effects and data state are correct.

Run demos — prove a business flow completes in the current environment; cannot prove exception, concurrency, and capacity boundaries hold.

Load testing and chaos drills — prove behavior meets targets under specific load and failure conditions; cannot prove all production risks are eliminated.

A batch import page showing "10,000 successful" only proves the page displays that number. Reconciliation with task records, database results, and failure details is needed to judge whether the business outcome truly holds. Verification conclusions must match the evidence level; local success cannot be directly extrapolated to production readiness.

Evidence Must Be Produced by Real Toolchains and Linked to Versions

An Agent writing "all tests passed" in its answer is not verification evidence.

CI provides a relatively independent, repeatable execution environment, but a passing pipeline is not universal proof. If tests are weakened in the same change, the pipeline faithfully executes a lowered standard. Therefore, key acceptance cases and gate rules need extra protection: they can be modified, but not by the current implementer to pass checks in the same execution path.

When Fixing Defects, Don't Lower the Acceptance Standard

Suppose an existing acceptance case requires that duplicate submission of the same business batch returns the original task instead of writing new data. The development Agent's implementation creates a new task, causing the test to fail. The correct fix is to modify the idempotency logic and re-run the original case. If the Agent simultaneously changes the test to "allow creating a new task", the test passes but the business risk is hidden.

Acceptance cases may be wrong, but modifying them requires independent justification, owner confirmation, and impact analysis. The code-fixing role can propose changes but cannot unilaterally alter the definition of done. For regression cases born from production incidents, the failure scene and expected result should be fixed first before allowing the development Agent to modify the implementation.

Design Documents Also Need Cross-Checking by Test Role

Independent verification cannot start only after code is done. In practice, the test role first checks whether the design is complete, consistent, and verifiable. For example, a batch import design states "failures can be retried"; the test role asks: which failures allow retry, is there an idempotency key, what state after retries exhaust, how does the user see the final result. If the design lacks verifiable conditions, even accurate development cannot yield trustworthy acceptance.

The test role produces two outputs:

A gap list against requirements and design baselines, fed back to architecture or business roles.

Verification evidence for code implementation and run results, fed back to development and release processes.

It does not replace the architecture role in modifying designs, nor should it fix business code for the development role to meet deadlines.

Continuous Eval Verifies Agent Behavior

As mentioned earlier, project specs, Skills, and Hooks are mutable software assets. After model, prompt, Skill, tool version, or permission changes, a stable set of tasks and hard cases must re-evaluate: whether the correct baseline is read, whether the Agent stops when design is missing, whether existing failure cases are preserved, whether real evidence is produced. This Eval checks for regression in the R&D Agent's working style; it cannot replace the batch import feature's own business tests and acceptance.

What Can Be Automated vs. What Requires Human Judgment

Build success, test exit code zero, interface structural compatibility, secret detection in files — these are suitable for automatic adjudication.

But the following still need explicit owners:

Whether partial success meets business and user expectations.

Whether current test data represents real risks.

Whether a remaining defect can be released conditionally.

Whether permissions and audit scope satisfy business accountability.

Whether failure impact has been accepted by users and business.

Humans need not repeat line-by-line machine checks; they should focus on semantics, risk, and authorization.

Summary

Agents running tests is a major productivity advance; but trustworthy verification cannot rest on the implementer's self-declaration. Self-verification drives local task convergence; independent verifier and test roles find gaps; CI produces repeatable evidence; business acceptance confirms real results. Together they form a verification chain that no single Agent can arbitrarily lower.

The next article will discuss what happens after verification passes: Agents can prepare merges and releases, but why they must not bypass production gates.