Why the Last Mile of AI Testing Requires Real‑Hardware Evidence

Reasoning Independence vs Evidence‑Source Independence

When an AI agent writes tests, triggers execution, and judges outcomes, a passing report can give the illusion that another intelligent entity has validated the result. The critical question is not the agent’s intelligence but the origin of the evidence it relies on.

Introducing a second model, independent reviewer, or agent without shared history provides reasoning independence : the new reasoner can spot intent bias, logical errors, or self‑consistent conclusions of the first agent.

However, if both agents depend on the same incomplete evidence, they inherit the same blind spot. When decisive evidence has not been generated, a stronger second agent cannot discover the defect merely by reasoning. The problem lies in the absence of evidence, not in the reasoning process. This distinction is called evidence‑source independence .

Out‑of‑Context Failures

Out‑of‑context failures are defects that appear only when an application runs on real hardware. No matter how wide a model’s context window or how capable the model is, it cannot fabricate this evidence because the execution has not produced any result to read. The limitation is about the source of evidence, not the rigor of reasoning.

Generation Loop (Kobiton MCP Server)

All failures stem from a single loop: an AI coding agent uses Kobiton’s Model Context Protocol (MCP) server to send a test‑run request to a real device, then receives session video, device logs, and captured states that the model did not write. The agent can request runs and read results, but it cannot generate evidence because the evidence is produced by the hardware’s response. The final conclusion depends on the actual run’s return.

Five Concrete Failure Categories

Cached Artifact State – The prototype build report shows a configuration error, yet every artifact the agent can read appears normal: correct code, loaded keys, and an inlined package that validates. The device silently runs an older cached package; no record on the phone shows which package actually resides. Forcing a stop and restart makes the device pull the current package, and the defect disappears without code changes. The cached build state lives on the device and expires there; without a run that creates this fact, nothing surfaces it.

Installation Source – Two environments contain byte‑identical binaries, yet one environment’s in‑app purchase works while the other’s does not. The binary itself cannot explain the difference; the OS’s determination of the app’s installation method influences how the OS validates the app‑store signature. A client reported that the device‑reported installed version differed from the actual released version in production, not in a lab environment. Installation source is a property of the device, not of the file.

Sensor and Host Dependency – The same APK, after reverse‑engineering and byte‑code verification, runs successfully on one Kobiton host but throws a connection error on another identical host. The discrepancy originates from the specific machine’s state, not from anything contained in the build. No artifact records the host’s state, so re‑reading the build cannot uncover the problem. Only a run on the faulty host can reveal it; a run on a successful host proves the build itself is fine.

Host State Drift Over Time – A host that previously passed the same build later fails without any code changes. A backend port migration caused two services to contend for the same port. The binary did not change, and the earlier pass cannot predict the later failure because the variable is the machine’s state at a later moment.

Contested Input Region – The app draws the Next and Calculate buttons above the system gesture navigation bar. Visually the buttons are on top, but the system treats taps in that area as a home‑button press because the OS reserves that region. On newer Android 16 devices the system reports the expected embedded region as zero. The model reviewing the layout code may note the missing inset, but the real issue is that the OS reserves the area, causing touch events to be interpreted as a system gesture.

Related Oversight: Missing Accessibility Checks

During a routine test session an automated accessibility scanner found 87 issues across 27 screens, while both manual tests and AI‑generated tests passed. The failures existed, but the evidence gap lay in coverage: neither the author nor the agent wrote assertions for accessibility, so the issues were missed. This is an attention‑deficit, not an evidence‑source defect, yet it has the same practical impact—real defects are marked as passed.

Why Device Data Cannot Be Replaced by Model Context

Screenshots, logs, videos, and local build hashes can be captured and fed into the model context, but the model cannot generate device data. All such data originates from a program that first runs on real hardware. The model cannot directly ingest three classes of information:

Facts that were never recorded because they never happened.

Results produced by a device that was never run, because the condition resides in the host’s state.

Signals from situations no one thought to check, because no capture was defined.

Capturing, executing, and focusing on device evidence is useful, but it only occurs after a physical run. The hardware always bears the load; the problem is not model intelligence but whether evidence was ever generated.

Evidence‑Source Control

Audit trails must satisfy integrity, traceability, retention, and segregation. An AI agent that writes code, writes tests, runs tests, and submits results inherently violates segregation because it creates and can modify all records, proving only work volume, not independent verification.

In regulated domains such as banking, medical devices, or gaming, sign‑off authorities must provide independent evidence of remediation; system‑generated records alone cannot serve as the sole basis.

Increasing Share of Source‑Side Failures

As model capabilities improve, intent mismatches and logical errors are increasingly caught by a stronger second agent, but source‑side failures may become a larger share of missed defects. Relying solely on better models for quality assurance still leaves the “last mile” problem.

Judgment Must Originate Outside the Model

If an AI agent misjudges a freshly reported test result, a second agent reasoning on the same data cannot help because neither agent has generated decisive evidence. Therefore, for out‑of‑context failures, the determination and its basis must originate from domains the model cannot influence: hardware it does not drive and artifacts it cannot create.