How BadBoy Browser (3500+ Stars) Is Redefining AI Crawling

Core Idea

The internet was built for browsers, yet AI agents traditionally try to access it via APIs that most sites lack. bb-browser (BadBoy Browser) flips this model by running an adapter inside the user's browser tab, using the existing cookies and fetch() calls so the site sees the request as coming from the logged‑in user.

"Your browser is the API. No keys, no bots, no traditional crawlers needed."

Conceptual Shift

Without bb-browser, crawling is a "protocol reverse‑engineering era"; with it, we enter the "runtime parasitic era" where the tool runs directly in the browser, avoiding the need to crack signatures or reverse‑engineer encryption.

Comparison with Traditional Solutions

Playwright/Selenium : headless, isolated browsers; no login state.

Standard crawler libraries : no browser, require cookie extraction.

bb-browser : uses your real Chrome, preserves logged‑in state, appears as the user, avoids anti‑bot detection, and lets the page handle complex authentication.

Applicable Scenarios

AI agent data acquisition – direct access to logged‑in sites.

Cross‑platform research – gather information from many platforms in one run.

Social media monitoring – real‑time Twitter, Zhihu, Bilibili feeds.

Financial data scraping – Snowball, Eastmoney live quotes.

Competitive analysis – automated monitoring across multiple sites.

Content aggregation – combine data from many sources into reports.

Quick Start

Installation

npm install -g bb-browser    # Note: latest version has a bug; install 0.10 instead

Basic Commands

bb-browser site update          # Update community adapters

bb-browser site recommend       # List recommended adapters

bb-browser site zhihu/hot        # Get Zhihu hot list

Common Command Examples

# Search tweets

bb-browser site twitter/search "AI agent"

# Zhihu hot list

bb-browser site zhihu/hot

# Search arXiv papers

bb-browser site arxiv/search "transformer"

# Real‑time stock quote

bb-browser site eastmoney/stock "Moutai"

# Search job postings

bb-browser site boss/search "AI engineer"

# Wikipedia summary

bb-browser site wikipedia/summary "Python"

# YouTube transcript

bb-browser site youtube/transcript VIDEO_ID

# StackOverflow search

bb-browser site stackoverflow/search "async"

Running the above on Zhihu hot topics yields screenshots (see images in the original article).

Platform Support

bb-browser provides commands for 36 platforms, covering search, social media, news, developer resources, video, entertainment, finance, recruitment, knowledge bases, shopping, and various tools. Each platform has a dedicated command set (e.g., search, feed, hot, stock, etc.).

OpenClaw Integration

When used with OpenClaw, bb-browser runs inside the built‑in browser without needing a Chrome extension or daemon:

bb-browser site reddit/hot --openclaw

bb-browser site xueqiu/hot-stock 5 --openclaw --jq '.items[] | {name, changePercent}'

OpenClaw skill name: bb-browser-openclaw.

MCP Integration (Claude Code / Cursor)

{
  "mcpServers": {
    "bb-browser": {
      "command": "npx",
      "args": ["-y", "bb-browser", "--mcp"]
    }
  }
}

Full Browser Automation

bb-browser also functions as a complete browser automation tool:

# Open a page

bb-browser open https://example.com

# Get accessibility tree

bb-browser snapshot -i

# Click an element

bb-browser click @3

# Fill an input

bb-browser fill @5 "hello"

# Run JavaScript

bb-browser eval "document.title"

# Authenticated fetch

bb-browser fetch URL --json

# Capture network requests

bb-browser network requests --with-body --json

# Screenshot

bb-browser screenshot

All commands support --json output, inline filtering with --jq <expr>, and concurrent tab operations with --tab <id>.

Adapter Complexity Levels

Level 1 – Cookie (direct fetch): Reddit, GitHub, V2EX (~1 min).

Level 2 – Bearer + CSRF token: Twitter, Zhihu (~3 min).

Level 3 – Webpack injection / Pinia store: Twitter search, Xiaohongshu (~10 min).

Tests show that 20 AI agents running in parallel can each reverse‑engineer a site and generate a usable adapter, driving the marginal cost of adding new sites for AI agents toward zero.

Architecture

AI Agent (Claude Code, Codex, Cursor, etc.)
      │ CLI or MCP (stdio)
      ▼
bb-browser CLI ──HTTP──▶ Daemon ──CDP WebSocket──▶ Your real browser
                              │
                              ┌───────┐
                              │ Per‑tab │
                              │ event cache │
                              │ (network, console, error) │
                              └───────┘

Daemon Configuration

The daemon binds by default to localhost:19824. You can change the host:

# IPv4 only (fix macOS IPv6 issue)
bb-browser daemon --host 127.0.0.1

# Listen on all interfaces (for Tailscale/ZeroTier remote access)
bb-browser daemon --host 0.0.0.0

Impact on AI Agents

Without bb-browser, AI agents are limited to files, terminals, and a few key‑protected APIs. With bb-browser, they gain direct website access, turning the workflow into files + terminal + websites. In under a minute an AI agent can perform cross‑platform research across arXiv, Twitter, GitHub, StackOverflow, Zhihu, and 36Kr.

Reference Resources

- GitHub repo: https://github.com/epiral/bb-browser
- npm package: https://www.npmjs.org/package/bb-browser
- Community adapters: https://github.com/epiral/bb-sites
- OpenClaw skill: https://clawhub.ai/yan5xu/bb-browser
- OpenClaw website: https://openclaw.ai