How WebBrain Puts an Open‑Source AI Agent Inside Your Browser Sidebar to Automate Online Tasks

What Is a “Browser AI Agent”

A browser agent is an AI program that can autonomously operate a webpage: given a natural‑language request such as “open GitHub and show today’s trending projects,” it opens the page, reads content, scrolls, clicks, and completes the task.

One‑Sentence Description

WebBrain is an open‑source AI browser‑agent extension that lives in the browser sidebar, uses the currently logged‑in session, and can read pages, answer questions, click, fill forms, and navigate—all while running the model of your choice.

Problems It Solves

Traditional browser automation suffers from three major issues:

Headless browser + script : No login state, breaks on captchas or SSO, and CSS‑selector scripts collapse when the page changes.

RPA tools : High onboarding cost, complex configuration, and still brittle when the UI is updated.

Pure LLM + screenshot : Limited resolution, unstable OCR, and fails on dynamic content.

WebBrain addresses these pain points with three key designs:

It runs inside your already‑logged‑in browser, instantly providing the required session.

It reads the accessibility tree instead of fragile CSS selectors, making it resilient to UI changes.

It is a multi‑step autonomous agent, deciding the next action based on real‑time feedback rather than a static script.

Three Permission Levels: Ask, Act, Dev

Ask (read‑only) : The agent can only read the current page and answer questions or extract URLs. No clicks or edits are performed.

Act (interactive) : The agent can click, type, scroll, upload/download, and fill forms. Before each critical action, a permission prompt asks the user to approve.

Dev (developer tools) : Builds on Act by exposing page source, styles, console, network requests, and reversible page edits, useful for debugging.

The three‑level model plus per‑action permission prompts balance AI autonomy with user control.

How It Works: Accessibility Tree Instead of Selectors

Instead of CSS selectors like #login-button, WebBrain reads the browser’s accessibility tree—a hierarchical, labeled text representation generated for screen readers. This lets the AI “see” the page structure as a document.

Workflow:

you enter a task ↓ agent reads the current page’s accessibility tree ↓ AI understands the structure and generates an operation plan (shown in the sidebar for approval) ↓ agent executes the plan step‑by‑step (click, fill, navigate) ↓ after each step AI receives feedback and decides the next move ↓ task completes or pauses for further confirmation

A “plan approval” step ensures the AI does not act blindly; the user must approve the structured plan before execution. The tool loop supports up to 130 steps by default (configurable to 195), with a “continue” button when the limit is reached.

Core Capabilities

Model‑agnostic : Supports any local model (llama.cpp, Ollama, vLLM) or cloud API (OpenAI, Claude, Gemini, DeepSeek, MiniMax, Kimi, Qwen, GLM). Includes 106 provider presets and a hosted default model called WebBrain Cloud.

Scheduling and watching : Commands /schedule to run a task later, and /watch to poll a page until a condition is met (e.g., price drops below a threshold), checking every 60 seconds after the initial run.

Workflow persistence : After a successful multi‑step run, /workflow --save stores it as a reusable, exportable workflow; /teach records a manual demonstration and turns it into a workflow.

MCP server : Runs npx -y @webbrain/mcp-server to expose WebBrain’s abilities to other coding agents (Claude Code, Codex, Cursor) via the webbrain_run command, allowing them to control the logged‑in browser.

Skills system : Packages trusted commands and tools as “skills” that load only when relevant, keeping the model’s context clean.

Typical Use Cases

Content Q&A and summarization on long articles, papers, or financial reports (Ask mode).

Form filling and repetitive button clicks (Act mode) with safety prompts.

Automation that requires a logged‑in session, such as checking GitHub notifications or exporting data from personal SaaS dashboards.

Timed monitoring and alerts, e.g., ticket availability or price tracking using /watch.

Providing a “hand” for coding agents that need to interact with a browser during tests or verification.

Security and Privacy

Permission gate : Every critical action requires explicit user approval; the default temperature is low (0.15) for predictable behavior.

Prompt‑injection defense : Web content is treated as untrusted data and never executed as a prompt.

Local‑first model option : When using an offline model, page content and conversation never leave the browser.

Accessibility‑tree reading : More stable and token‑efficient than screenshot OCR.

Browser support note : Firefox lacks full Chrome DevTools Protocol features, so some interactions (shadow‑DOM, real mouse events) are weaker compared to Chrome.

Comparison with Alternatives

Headless browsers (Playwright, Puppeteer) : Fully controllable but lack login state and rely on fragile selectors; WebBrain combines AI understanding with a real logged‑in browser for personalized tasks.

Traditional RPA tools : Good for fixed, high‑volume enterprise workflows but require heavy configuration and break on UI changes; WebBrain’s AI‑driven adaptation lowers the barrier.

Other AI browser agents (closed‑source) : Often lock the model and backend; WebBrain is MIT‑licensed, self‑hostable, model‑agnostic, and provides an MCP interface for extensibility.

Selection guide :

Need AI‑driven web tasks with login state and optional local model → WebBrain.

Enterprise batch automation → traditional RPA.

Fine‑grained developer control → Playwright‑style libraries.

Who Should Use It

Everyday users and productivity enthusiasts who want AI‑assisted browsing without coding.

Developers who want to integrate browser actions into coding agents or debug pages via Dev mode.

Privacy‑conscious users who prefer offline models.

Not ideal for one‑off simple Q&A (use ChatGPT), Firefox‑only users, or fully unattended high‑risk operations.

Design Lessons

Embedding the agent in the user’s real environment avoids the complexity of simulating login, captchas, and anti‑scraping measures.

Reading the accessibility tree yields stable, semantic page understanding compared to brittle selector matching.

Granular permission levels and pre‑execution plan approval keep AI powerful yet safe.

Generating a plan before acting is more reliable than “act‑then‑think.”

Providing an MCP interface lets the agent become a reusable component for larger AI ecosystems.

If you’re frustrated that AI can code but can’t browse the web for you, WebBrain puts the browser directly under the AI’s control while preserving your login state and safety.

WebBrain – an open‑source AI agent that lives in the browser sidebar.

GitHub address: github.com/webbrain-one/webbrain