33 Open-Source Web Crawlers to Supercharge Your Data Collection

Java Crawlers

Arachnid – A lightweight Java web spider framework with a simple HTML parser; GPL license; micro‑spider framework.

crawlzilla – Java‑based search engine builder integrating Nutch, offering HTML and document crawling, Chinese word segmentation; Apache License 2; Linux.

Ex‑Crawler – Java web crawler with daemon process and database storage; GPLv3; cross‑platform.

Heritrix – Modular Java crawler with strong extensibility; Apache license; cross‑platform; respects robots.txt.

heyDr – Lightweight, multithreaded vertical search crawler; GPLv3; cross‑platform.

ItSucks – Java web spider with Swing GUI for rule definition; GPL license.

jcrawl – Small, high‑performance Java spider; Apache license; cross‑platform; extracts various file types.

JSpider – Configurable Java spider executed via command line; LGPL license; cross‑platform; highly extensible.

Leopdo – Java web search and crawler with full‑text and vertical search capabilities; Apache license; cross‑platform.

MetaSeeker – Complete web content extraction, formatting, integration, and storage solution; GPL license; supports server‑side and client‑side crawling.

Playfish – Java‑based highly customizable web crawler using XML configuration; MIT license; cross‑platform.

Spiderman – Micro‑kernel + plugin architecture Java crawler; Apache license; cross‑platform; no code required.

webmagic – Zero‑configuration Java crawler framework with modular design, multithreaded and distributed support; Apache license; cross‑platform.

Web‑Harvest – Java web data extraction tool using XSLT, XQuery, and regex; BSD license; cross‑platform.

WebSPHINX – Java interactive development environment for web crawling; Apache license; cross‑platform.

YaCy – P2P‑based distributed web search engine and crawler; GPL license; Java/Perl; cross‑platform.

Python Crawlers

QuickRecon – Information‑gathering tool for subdomains, emails, and relationships; GPLv3; Windows/Linux.

PyRailgun – Simple, lightweight Python crawler supporting JavaScript‑rendered pages; MIT license; cross‑platform.

Scrapy – Asynchronous Twisted‑based Python crawler framework; BSD license; cross‑platform; extensive documentation.

C++ Crawlers

hispider – High‑performance C/C++ spider with URL deduplication, async DNS, distributed downloading; BSD license; Linux.

larbin – Open‑source C/C++ web spider for URL expansion; GPL license; Linux; high throughput.

Methabot – Optimized, highly configurable web/FTP/file‑system crawler; unknown license; Windows/Linux.

C# Crawlers

NWebCrawler – Configurable C# web crawler with thread control, MIME filtering, and statistics; GPLv2; Windows.

Sinawler – First Chinese Weibo crawler; .NET 2.0; uses SQL Server; GPLv3; Windows.

spidernet – Multi‑threaded recursive‑tree C# crawler storing data in SQLite; MIT license; Windows.

Web Crawler – Java‑based framework (listed under C# section) with Lucene integration; LGPL license; cross‑platform.

网络矿工 – .NET open‑source web data collector; BSD license; Windows.

PHP Crawlers

OpenWebSpider – Multi‑threaded PHP spider with many features; unknown license; cross‑platform.

PhpDig – PHP web crawler and search engine supporting PDF, Word, Excel, PowerPoint indexing; GPL license; cross‑platform.

ThinkUp – Social media data collector for Twitter, Facebook, etc.; GPL license; cross‑platform.

微购 – Open‑source shopping system built on ThinkPHP, also used for data collection; GPL license; cross‑platform.

Erlang Crawlers

Ebot – Scalable distributed Erlang web crawler with RESTful URL queries; GPLv3; cross‑platform.

Ruby Crawlers

Spidr – Ruby library to fully download websites or links locally; MIT license; cross‑platform.

Source: 36大数据