How One Developer Built a $10, 1 GB Search Engine to Rival Google

Project Overview

Marlin is an open‑source tool that builds a personal search index by crawling only homepage URLs. Using a cloud GPU rented for roughly $10 and less than 1 GB of disk space, the author catalogued about 560,000 homepages.

Architecture

The system consists of four processes:

Domain fetcher – retrieves a list of domains to crawl.

Executor – calls a small OpenAI‑compatible language model to read each page and generate a name, a two‑to‑three‑sentence summary, a category, and a few tags.

Manager – filters out undesirable pages before they enter the index.

API with Web UI – provides monitoring, search, and filter capabilities.

Initial Run and Weighting System

The first version was operational within a few hours. When pointed at a sample domain set, the crawler quickly aggregated content, but more than 90 % of the results were corporate sites and documentation, not the intended personal content. To mitigate this, a weighting system was added that pushes preferred pages to the top of the crawl queue and demotes less relevant ones, rather than blocking domains outright.

Crawl Termination and Cost

After crawling roughly 560 k pages, the crawl stopped because the prioritized categories had been exhausted and the system began pulling generic “raw internet” content. Total GPU usage cost was about $10.

Labeling and Categorisation Challenges

Allowing the language model to invent its own categories and tags resulted in 671 distinct categories—most appearing only once—and over 121,000 tags, with more than half used only a single time. A manual merging tool was built to clean up this chaos, and the author noted that the free‑form approach may not scale beyond hobby projects.

Repository and Documentation

The code is released under an open‑source license at https://github.com/alexmorleyfinch/marlin. The repository includes extensive operational guides, instructions for creating custom weight lists, and options to rent a cloud GPU if local hardware is insufficient.