Continuous GUI Verification: Ensuring Users Complete Tasks, Not Just Open Pages

An AI agent may generate a page that looks complete — buttons, tables, and success toasts all appear in screenshots — but real usage reveals failures: the second file upload errors, a refresh loses results, and a different account sees unauthorized data. The previous article examined code-level issues; this one shifts perspective to the user's actual workflow. GUI verification must walk the full operation flow.

Using customer data import as an example: upload file, view validation results, confirm submission, wait for task completion, then download error details. Buttons exist, yet the steps may not connect smoothly.

Define What "Done" Looks Like

"Verify the import page" is too vague; an agent might interpret completion as merely opening the page and clicking a button. Instead, specify concrete acceptance criteria: prepare a file with both valid and invalid rows; after submission, valid rows are written per rules, invalid rows are downloadable; refreshing the page retains the batch result; resubmitting the same batch does not duplicate processing. At this specificity, the agent knows what to check. Playwright's best practices also emphasize testing user-visible behavior rather than binding tightly to page internals.

Page toasts are evidence but cannot substitute business results. A toast saying "import complete" may appear while the database is still asynchronously writing. Conversely, a network hiccup may prevent the page from receiving a response even though the backend has finished.

Break Scenarios Down to Pinpoint Issues

Don't let the agent freely explore the entire system at first. Convert key user flows into repeatable verification scenarios first, then use exploration to fill gaps.

Valid file submission : UI shows batch progress and result; cross-check that corresponding batch produces expected records.

File contains error rows : UI shows error locations and reasons clearly; cross-check that valid rows, error rows, and total count match.

Refresh or re-login : UI can rediscover existing task; cross-check that task identity and result remain unchanged.

Duplicate submit clicks : UI behavior is explicit, no duplicate task confusion; cross-check that backend does not write duplicates.

Regular user attempts privileged action : UI shows or denies per permissions; cross-check that server still rejects directly crafted requests.

UI tests need not cover all low-level checks. API, permission, and data consistency tests can complement; the key is linking these evidences with the same batch or business identifier.

Test Interference Makes Diagnosis Hard

Two tests sharing one account, one file, and one batch can overwrite each other. A pass today and a fail tomorrow may not indicate a product regression — environment residue could be the cause. Therefore, each scenario must have an isolated data scope: prepare before run, clean up after per agreement. This aligns with Playwright's recommendation for test isolation. Data setup can use test APIs, but the actual user submission step must still be driven from the page, not bypassed for convenience.

If a dependency is mocked, note it in the results. Mocking upload success can verify subsequent page behavior but cannot prove real file storage is wired. Real dependency validation deserves a separate environment and test cases. Likewise, avoid defaulting all scenarios to an admin account; identity differences are conditions that must be covered.

Agent Explores, Stable Paths Must Be Replayable

Letting the agent operate freely helps discover missing entry points, odd interactions, and copy ambiguities. But relying on its ad-hoc path selection every run makes failures hard to reproduce. For confirmed critical flows, I prefer preserving them as repeatable automated tests. Locate a button by its role and accessible name (e.g., "Confirm Import") rather than its DOM depth. Wait for the batch result to appear instead of sleeping a fixed number of seconds.

Async tasks especially must avoid indefinite waits. Beyond the agreed timeout, retain the batch state, network requests, and page context to determine whether the failure is a product defect, environment issue, or test flaw.

If a rerun passes, the first failure cannot be erased. Playwright distinguishes first-run passes from retry passes — a useful distinction: retries help gather clues but do not automatically eliminate flakiness.

After Failure, Can You See Where It Stuck?

A verification record can include application version, scenario and data version, account role, execution time, assertion results, and failure trace. Browser traces help reconstruct clicks, requests, and page changes; screenshots suit layout and error toast illustration.

Not every run needs a full video. The artifacts must map to the specific batch and show exactly which step failed to have diagnostic value. Traces and screenshots may capture customer data, login credentials, or tokens; collection must consider masking, access control, and retention policies.

Continuous verification does not mean running every browser and device on every commit. Check high-frequency paths on commit, expand coverage before release, and combine with risk-based real-environment observation. Coverage trade-offs should be documented to avoid extrapolating success on one browser to all devices.

The desired verification outcome is concrete: the user can finish the task and backend data matches; when they cannot, the page tells them where they are stuck and what to do next. Screenshots assist but cannot stand in for these results.

One more boundary to examine next: when the agent reads operational requirements from the page or attachments, what decides whether it follows them?

References

Playwright, Best Practices: https://playwright.dev/docs/best-practices

Playwright, Retries: https://playwright.dev/docs/test-retries