A Free, No‑API‑Key Agent Web Toolkit: Local Search, Deep Crawling, and Site Monitoring in One Package

What is Wigolo

Wigolo is a local networking toolbox for AI‑agents built around MCP‑Server . It provides search, page fetch, site crawl, structured extraction, local caching, research reporting and web‑monitoring entirely on the developer’s machine.

Core Capabilities

1. Parallel Engine Search with Evidence Tracing

A single search call queries 18 public search engines (including Bing, DuckDuckGo, Brave). The system deduplicates results, re‑ranks them with a local model, and returns full markdown pages. Each result includes:

Original excerpt with byte‑level source position

Complete markdown content (no snippet limit)

Quality score, reference ID and engine status

Explicit failure flags for engines that returned low‑quality or error results

Byte‑level evidence enables the downstream LLM to locate the exact origin of each answer, reducing hallucinations.

2. Intelligent Page Fetch

The fetch command uses a layered strategy:

Lightweight HTTP request for static pages.

If the page is a SPA, behind anti‑scraping walls, or requires rendering, it falls back to a built‑in headless browser.

Cookies are preserved across requests to speed up subsequent accesses.

Supports PDF parsing and interactive actions (click, input, scroll, screenshot).

On failure, returns a clear error code such as blocked_by_challenge. Cloudflare challenges may require a manual proxy configuration.

3. Full‑Site Crawl & Structured Extraction

Two commands support site‑wide operations: crawl – breadth‑first, depth‑first or sitemap mode; obeys robots.txt and automatically rate‑limits per domain. extract – automatically detects tables, articles, JSON‑LD metadata; can also accept a custom JSON‑Schema to extract specific fields.

4. Local Knowledge‑Base Cache

All fetched pages are stored under ~/.wigolo/. The cache provides both keyword lookup and semantic vector lookup, so repeated queries are served instantly without network traffic.

5. Research & Agent Automation

research

splits a complex question into sub‑questions, runs parallel searches, gathers evidence and produces a cited report (requires a local or cloud LLM). agent runs a fully automated fetch‑extract‑plan loop with an optional timeout.

6. Web Monitoring (diff / watch)

diff

compares two fetches and highlights changed sections. watch polls a page at intervals and pushes change notifications via webhook, useful for tracking documentation updates.

Installation Guide

Prerequisites: Node.js ≥ 20 and ~1.5 GB disk space for the headless browser engine and local vector model.

One‑click initialization: npx wigolo init – downloads the browser engine and model, validates components.

Optional MCP binding (e.g., Claude‑Code, Cursor): npx wigolo init --agents=claude-code,cursor.

Health check: npx wigolo doctor – inspects data directory, engine, model, LLM provider and suggests fixes.

Start the local REST service: wigolo serve – exposes the full search‑fetch API.

Uninstall cleanly: npx wigolo config --uninstall --yes – removes all files.

Limitations (Beta)

Beta stability; complex anti‑scraping scenarios may perform worse than mature paid services.

Consumes ~1.5 GB disk and CPU resources for headless rendering.

Public search engines can rate‑limit; parallel fallback reduces but does not eliminate the issue.

AGPL‑3.0 license requires derivative services to be open‑sourced.

Target Users

Developers who need up‑to‑date official documentation in Claude‑Code or Cursor.

Technical users building local agents or AI workflows who want to avoid paid cloud search APIs.

Teams that require long‑term monitoring of webpages or technical documentation.

Privacy‑sensitive users who do not want query data sent to third‑party servers.

Key Comparison Points

Deployment : Local execution vs. cloud API for competitors.

API Key : No key required for Wigolo; competitors need paid keys.

Cost : Unlimited free usage for Wigolo; others charge per request or have limited free tiers.

Data Flow : All data stays on the machine with Wigolo; competitors send data to external servers.

Evidence Location : Byte‑level source positioning in Wigolo; only summary links in other tools.

Headless Rendering : Built‑in in Wigolo; not available in some competitors.

Local Cache : Semantic cache in Wigolo; cloud cache or none elsewhere.

Web Monitoring : Supported (diff/watch) only in Wigolo.

MCP Integration : Native support in Wigolo; requires wrappers for others.

Observed Behaviour

In a side‑by‑side test using the same Claude‑Code session, Wigolo, Claude’s built‑in WebSearch, Tavily and Exa returned consistent core conclusions and authoritative sources. Wigolo was the only tool that provided precise byte‑level citations for each answer.

Repository

https://github.com/KnockOutEZ/wigolo