Why Scaling Web Crawlers Is Harder Than You Think: Lessons from 1,000B Pages

Web crawling may seem easy, but scaling it to extract data from thousands of e‑commerce sites presents a set of unique challenges. Scrapinghub, the creator of the popular Scrapy framework, shares hard‑earned lessons after crawling more than one trillion product pages since 2010.

Why Scalable Crawling Matters

When crawling at scale, speed and data quality become critical constraints. High‑throughput extraction must be fast without sacrificing the accuracy of the collected data.

Challenge #1 – Ever‑Changing Site Formats

E‑commerce sites often have sloppy, constantly evolving HTML, JavaScript, and API implementations. Issues include misused HTTP status codes, broken JavaScript, and improper JSON escaping, which can cause 404 pages to return 200 responses, require regex workarounds, or force full page rendering.

When a product is removed, the site may return a 200 status for a 404 page after an upgrade. Incorrect JSON escaping can break JavaScript, forcing regex extraction. Misused Ajax calls may require rendering the page or mimicking API calls, increasing development effort.

These problems make visual or automated extraction tools ineffective, and they multiply when dozens or hundreds of sites change every few months.

Challenge #2 – Scalable Architecture

A serial crawler that processes one request every 2–3 seconds cannot handle the millions of daily requests needed for large‑scale product extraction. A distributed, high‑throughput architecture is required, often separating product discovery crawlers from product extraction crawlers and allocating resources accordingly (e.g., one extraction crawler per 100,000 pages).

Challenge #3 – Maintaining Throughput Performance

Optimizing request latency and minimizing unnecessary requests are essential. Best practices include avoiding headless browsers unless absolutely necessary, reusing data from catalog pages instead of visiting each product page, and skipping image downloads unless required.

Use headless browsers like Splash or Puppeteer only as a last resort; they dramatically increase resource consumption. If needed data is available on the listing page, do not request the product detail page. Avoid downloading images unless unavoidable.

Challenge #4 – Anti‑Bot Countermeasures

Large e‑commerce platforms employ sophisticated anti‑bot solutions (e.g., Distil Networks, Incapsula, Akamai) that detect automated traffic via JavaScript challenges, IP rate limiting, and other techniques. Robust proxy management—rotating IPs, handling rate limits, and session management—is essential, often outsourced to specialized proxy providers.

Challenge #5 – Data Quality

At scale, manual validation is impossible. Automated QA pipelines, peer‑reviewed code, and machine‑learning‑based validation are needed to detect data type mismatches, product attribute inconsistencies across locales, and sudden drops or spikes in record counts caused by site changes.

Scrapinghub has built a monitoring system that flags validation errors, product attribute anomalies, and structural site changes, alerting teams to intervene promptly.

Summary

Scaling product data extraction involves tackling volatile site structures, building resilient infrastructure, preserving high throughput, bypassing anti‑bot defenses, and ensuring data integrity. The insights shared here aim to guide engineers in designing robust, scalable crawlers for massive e‑commerce datasets.