Enabling Agents to See the Browser: A Runtime Harness Using Chrome DevTools Protocol

Why Browser Use

When delivering web products the final artifact is the rendered page, not the source code. Static analysis and type checking cannot guarantee that the UI is correct, that there are no console errors, or that asynchronous state does not break the layout. Many defects only appear at runtime, such as slow API responses, container width changes, overflow clipping, or stale navigation states. Because an agent only sees code, it cannot detect these issues without a way to observe the browser.

Our Solution

We built and open‑sourced a set of Skills that give an agent programmable access to a real Chrome instance via the Chrome DevTools Protocol (CDP). The repository is available at:

https://github.com/hixuanxuan/browser-automation

The core skill visual‑verify lets an agent open a browser, check element existence, layout overflow, interaction outcomes, console messages and network requests, and finally produce a screenshot‑based report with assertions.

npx skills add hixuanxuan/browser-automation --skill visual-verify

What to Verify

We identified six verification dimensions:

Path : Can the agent navigate to the target page without errors?

Content : Are all expected elements present and contain the right text?

Visual : Does the layout match the design (no mis‑alignment, overflow, truncation)?

Interaction : Do clicks, form submissions and tab switches behave as intended?

Console : Are there any JS errors, React warnings or failed resource loads?

Network : Do backend requests return the expected status codes and payloads?

How to Set Up

CDP is the bridge that allows external programs to control Chrome. Launch Chrome with the remote‑debugging flag: --remote-debugging-port=9222 Three deployment modes are supported:

UI Chrome – visible window for interactive debugging.

Headless Chrome – runs without a UI, suitable for CI.

NoVNC – Chrome runs inside a container and is exposed via a web‑based remote desktop.

The agent also needs to start the development server automatically. Typical cues are the start command (e.g., npm run dev) and a health check such as the appearance of http://localhost:<port> in the output.

Core CDP Capabilities Used

Page navigation & waiting for elements.

DOM query & manipulation (click, fill, evaluate JavaScript).

Full‑page or element‑level screenshot.

Console listening via Runtime.consoleAPICalled and Runtime.exceptionThrown.

Network interception to capture request/response details.

Script injection for custom visual overlays or measurement.

Contract‑Driven Workflow

Verification is expressed as JSON contracts that list checkpoints, actions and assertions. A typical checkpoint looks like:

{
  "id":"CP-2",
  "desc":"Panel expand/collapse",
  "steps":[
    {"desc":"Click quick‑phrase button", "action":{"type":"click","selector":".quick-phrase-btn"},
     "assertions":[{"id":"C1","type":"visible","selector":".quick-phrase-panel","desc":"Panel visible"}]},
    {"desc":"Click again to hide", "action":{"type":"click","selector":".quick-phrase-btn"},
     "assertions":[{"id":"C3","type":"custom","desc":"Panel not visible","script":"return {pass: !document.querySelector('.quick-phrase-panel')?.offsetParent, reason: 'panel visibility'}"}]}
  ]
}

Before execution the contract is linted ( contract‑lint) and then run in a real browser with dom‑assert. Results are recorded in contract.md as a structured acceptance log.

Optimization Tips

Screenshot operations are token‑expensive for multimodal models. Use screenshots only for layout or visual regression checks; use DOM queries for precise style reads, form filling or existence checks. Annotated screenshots ( annotate‑screenshot.mjs) add bounding‑box measurements to guide the model. The experimental Visual Element Tree (VET) overlay paints semantic color blocks on the page; diffing VET images quickly reveals size or missing‑element issues. Maintain a visual‑notes.md file that records reliable selectors, failed selectors and timing cues to reduce repeated trial‑and‑error.

Practical Schemes

Scheme 1 – Heavy QA : After the main agent finishes a feature, a dedicated sub‑agent browser‑ui‑test‑inspector runs a full verification suite, producing a report.md with PASS/FAIL status, screenshots and reproducible steps. Issues are fed back to the main agent for fixing. This yields high confidence but incurs higher token and compute cost.

Scheme 2 – Light‑weight “Test While You Code” : The main agent writes a checkpoint after each meaningful change and immediately runs dom‑assert.mjs. Failures are fixed on the spot, and the agent appends observations to visual‑notes.md. This mirrors the typical “save → refresh → glance” loop but fully automated, offering lower cost and faster feedback.

Conclusion

Runtime visibility is a crucial component of an agent’s harness. CDP‑based scripts, screenshot and annotation tools, VET overlays, and a contract‑driven verification framework form reusable infrastructure that can be shared across projects. The main remaining challenge is cost: visual verification consumes significant model budget, so ongoing research must balance screenshot frequency, smarter DOM‑first strategies and more compact visual representations.