wigolo: Zero‑Key, Zero‑Cost Local‑First Web Search Engine for AI Coding Agents

Background

AI coding agents such as Claude Code and Cursor rely on web search and content extraction. Existing solutions suffer from three core problems:

Search cost : most services charge per query, and cost grows with usage.

Data privacy : queries and fetched pages pass through cloud services, risking leakage.

Duplicate queries : repeating the same question incurs additional cost.

wigolo adopts a local‑first principle: web search results are cached locally so each query becomes knowledge accumulation rather than a billable event.

Architecture & Principle

wigolo runs as a local MCP server and communicates with AI agents via the stdio protocol. The data flow is:

AI Agent → MCP protocol → wigolo server → local search / crawl engine
                ↓
        local cache (SQLite + vector index)
                ↓
        structured evidence → AI agent composes final answer

Search Engine

wigolo includes a parallel aggregation engine that queries 18 public search engines (or an optional self‑hosted SearXNG instance). Results are returned as structured evidence with a score breakdown (semantic, lexical, engine consensus, freshness) and are re‑ranked by an ML‑driven pipeline.

Crawl Engine

The crawler uses a three‑layer progressive strategy:

Plain HTTP : fast path covering most sites.

TLS fingerprint simulation : mimics real browsers to bypass bot detection.

Headless browser (Playwright) : renders full JavaScript pages when earlier layers fail.

Each layer automatically upgrades based on signals such as SPA markers, captcha prompts, or thin content.

Local Cache

All search results and fetched pages are stored in a SQLite database under ~/.wigolo/, with full‑text, keyword, and vector indexes. Repeated queries hit the cache and return in milliseconds.

Core Features

Local Search Engine : 18‑engine parallel aggregation, ML‑driven re‑ranking, per‑result score breakdown, structured evidence ready for AI consumption.

Intelligent Crawl Engine : automatic SPA and captcha detection, CSS selector / table / metadata extraction, clean Markdown output after Readability and defuddle processing.

Persistent Local Cache : hybrid keyword + semantic vector search, change detection with diff summaries, similar‑page recall from cached content, all data stays on the local machine.

Deep Research Pipeline : automatic problem decomposition into sub‑queries, parallel fan‑out search, source aggregation, citation‑rich research brief, three depth levels (quick, standard, comprehensive).

Autonomous Data‑Collection Agent : AI‑driven planning of URLs, parallel fetching, optional JSON‑Schema extraction, page‑budget and time‑budget constraints, detailed step‑by‑step logs.

Practical Scenarios

Scenario 1 – Retrieve latest technical documentation

Claude Code calls wigolo’s search tool via MCP.

wigolo parallel‑searches 18 engines and caches the results.

Subsequent identical queries hit the local cache and return instantly.

Claude Code synthesizes the final answer from the structured evidence.

Scenario 2 – Full‑site crawl for documentation analysis

Invoke wigolo’s crawl tool with a seed URL.

wigolo traverses the site using BFS/DFS/site‑map strategies.

Each page is cleaned with Readability and stored as Markdown.

Cached data can be queried later without additional network requests.

Scenario 3 – Continuous page‑change monitoring

Initial fetch caches the page content.

Subsequent fetch calls automatically detect changes.

wigolo returns a diff summary of the modifications.

Installation

Environment Requirements

Node.js >= 20

~1.5 GB free disk space

macOS / Linux / Windows

Install Command

# One‑click install and configure for specific AI agents
npx wigolo init --non-interactive --agents=claude-code,cursor
# Verify installation
npx wigolo doctor

Manual MCP Configuration

{
  "mcpServers": {
    "wigolo": {
      "command": "npx",
      "args": ["-y", "wigolo"]
    }
  }
}

Optional Answer Synthesis (LLM provider)

# Use Gemini
export WIGOLO_LLM_PROVIDER=gemini
export GEMINI_API_KEY=<your_key>
# Or use local Ollama (offline)
export WIGOLO_LLM_PROVIDER=ollama
export WIGOLO_LLM_MODEL=llama3.1

Hands‑On Examples

Example 1 – Claude Code integration

# Install for Claude Code
npx wigolo init --non-interactive --agents=claude-code
npx wigolo doctor
# Ask a question; wigolo supplies web results
claude "What are the new logical replication features in Postgres 17?"

Example 2 – Research tool

claude "Use wigolo research to study the latest trends in Rust async runtimes"

The research tool performs:

Automatic decomposition of the question into sub‑queries (e.g., "Rust async runtimes comparison", "tokio vs async‑std", "latest developments").

Parallel search for each sub‑query.

Fetching of source pages.

Synthesis of a citation‑rich research brief.

Example 3 – Full‑site crawl

claude "Use wigolo crawl https://docs.example.com to analyze all API endpoints"

The crawl tool traverses the site, extracts clean Markdown content, and caches it locally.

Comparison with Known Projects

Query cost : wigolo – no cost; Firecrawl, Exa, Tavily – pay‑as‑you‑go.

Local‑first : wigolo – fully local; others – cloud‑based.

Multi‑engine search : wigolo – 18 engines; Firecrawl/Exa/Tavily – support aggregation.

Full‑site crawl : wigolo – supported; Exa – not supported.

Structured extraction : wigolo – CSS, tables, metadata, JSON‑LD; others – basic extraction.

Local cache : wigolo – SQLite + vector index; others – none.

Change detection : wigolo – supported; others – not supported.

Similar‑page recall : wigolo – local + web supplement; Exa – limited.

Deep research : wigolo – problem decomposition + synthesis; others – not supported.

Autonomous agent : wigolo – planning, search, extraction; others – not supported.

Data privacy : wigolo – data never leaves the machine; others – data passes through cloud.

Offline query : wigolo – cache hits work offline; others – require network.

License : wigolo – AGPL‑3.0; others – proprietary.

References

Project repository: https://github.com/KnockOutEZ/wigolo
Official documentation: https://knockoutez.github.io/wigolo/

Disclaimer: Content is for learning and exchange only. Use the tools in compliance with platform terms and applicable laws; refer to official sources for the latest policies.