Scrapy vs. Gevent: Choosing the Right Python Web‑Crawling Framework

The author, a Python developer with three years of experience, shares a candid overview of web‑crawling tools after a recent job search, focusing on Scrapy and gevent‑based solutions.

Scrapy Overview

Scrapy 0.16, released in early 2013, had become a mature framework after a year of development. By version 0.18 and 0.20, features such as HTTP keep‑alive, RFC‑compliant cache, DNS caching, and URL deduplication were added. The framework is praised for its completeness and rapid custom development.

Comparison with Other Crawling Options

Java : Nutch, Heritrix – good for generic crawlers, abundant documentation, but heavy for highly customized spiders.

Ruby : Exists a crawling library, but details are scarce.

Node.js : Some startups use it; the author has limited exposure.

Python (gevent, others) : Discussed in detail later.

Using gevent for Crawling

Typical stack: gevent + requests + a queue (Redis, Beanstalk, etc.). Key caveats:

After monkey_patch, many debugging tools stop working.

Requests determines encoding from HTTP headers, ignoring the page‑declared charset, which can cause mis‑decoding.

gevent’s synchronization primitives are limited; performance may drop in complex scenarios (e.g., an HTTP‑proxy scheduler).

Other details: gzip handling, connection pooling, etc.

Advantages of gevent over Scrapy:

Lower‑level customization is simpler because it uses plain sockets instead of Twisted.

For a few lightweight spiders, gevent can be more straightforward and faster for pure downloading when bandwidth is abundant.

Practical Tips and Common Issues

Understand Scrapy’s built‑in features (URL deduplication, DNS cache, keep‑alive, gzip) to avoid reinventing the wheel.

JavaScript rendering : Use Selenium, PhantomJS, CasperJS, Ghost, WebKit, ScrapyJS, or Splash. ScrapyJS combines WebKit’s event loop with Twisted for fully asynchronous JS handling.

Content extraction : XPATH, PyQuery, CSS selectors; for plain‑text extraction, Scrapy can expose a plain_txt endpoint.

Fuzzy matching & ML : Scrapy’s GitHub repos contain libraries for fuzzy matching and machine‑learning‑based parsing, though results vary.

Distributed crawling : Split tasks by target, run spiders on separate machines, or replace the queue for a peer‑to‑peer model. Existing open‑source implementations are on GitHub.

Deployment & scheduling : Scrapy recommends scrapyd. Large‑scale scheduling still requires custom development (e.g., storing spider configs in a DB and providing a web UI for start/stop).

IP blocking : Purchase bulk IPs cautiously; free proxies are unreliable. Implement a proxy‑rotation scheduler; Scrapinghub’s Crawlera offers a commercial solution, and the author built a local equivalent.

URL deduplication at scale : For tens of millions of URLs, examine Scrapy’s deduplication logic and consider Bloom filters (memory formula provided in Scrapy’s docs).

Storage : MongoDB is common; for advanced needs, HBase is suggested (see Scrapinghub blog post).

Hardware limits : On low‑end machines (e.g., i3, 4 GB RAM), disk I/O and memory become bottlenecks; over‑large queues can cause cascading slowdowns.

Monitoring : scrapyd offers basic monitoring; more detailed metrics require a custom web service.

Resources and Tools

Scrapy GitHub organization: https://github.com/scrapinghub

ScrapyJS & Splash for JS rendering.

WebStruct and other ML‑related repos under the same organization.

Scrapy’s SEP (similar to PEP) and official documentation.

Scrapinghub blog: http://blog.scrapinghub.com/

Scrapy wiki: https://github.com/scrapy/scrapy/wiki

Deployment Recommendations

Use scrapyd for simple deployment, as recommended by the Scrapy project. For large‑scale distributed crawling, replace the default scheduler with a persistent queue (Redis, Beanstalk, etc.) and consider building a custom web dashboard to manage spider lifecycles.