Build a Distributed Scrapy Crawler in Minutes with RabbitMQ and RedisBloom

Learn how to quickly develop a distributed Scrapy crawler using the Scrapy-Distributed extension, RabbitMQ, and RedisBloom.

Quick Start

Step 0: Install Scrapy-Distributed pip install scrapy-distributed If you lack the required services, start RabbitMQ and RedisBloom containers for testing:

# pull and run a RabbitMQ container.
docker run -d --name rabbitmq -p 0.0.0.0:15672:15672 -p 0.0.0.0:5672:5672 rabbitmq:3
# pull and run a RedisBloom container.
docker run -d --name redis-redisbloom -p 0.0.0.0:6379:6379 redislabs/rebloom:latest

Step 1 (optional): Create a spider project scrapy startproject simple_example Modify the spider file, for example:

from scrapy_distributed.spiders.sitemap import SitemapSpider
from scrapy_distributed.queues.amqp import QueueConfig
from scrapy_distributed.dupefilters.redis_bloom import RedisBloomConfig

class MySpider(SitemapSpider):
    name = "example"
    sitemap_urls = ["http://www.people.com.cn/robots.txt"]
    queue_conf: QueueConfig = QueueConfig(
        name="example", durable=True,
        arguments={"x-queue-mode": "lazy", "x-max-priority": 255}
    )
    redis_bloom_conf: RedisBloomConfig = RedisBloomConfig(key="example:dupefilter")

    def parse(self, response):
        self.logger.info(f"parse response, url: {response.url}")

Step 2: Edit settings.py to use the distributed scheduler and dupefilter

SCHEDULER = "scrapy_distributed.schedulers.DistributedScheduler"
SCHEDULER_QUEUE_CLASS = "scrapy_distributed.queues.amqp.RabbitQueue"
RABBITMQ_CONNECTION_PARAMETERS = "amqp://guest:guest@localhost:5672/example/?heartbeat=0"
DUPEFILTER_CLASS = "scrapy_distributed.dupefilters.redis_bloom.RedisBloomDupeFilter"
BLOOM_DUPEFILTER_REDIS_URL = "redis://:@localhost:6379/0"
BLOOM_DUPEFILTER_REDIS_HOST = "localhost"
BLOOM_DUPEFILTER_REDIS_PORT = 6379
REDIS_BLOOM_PARAMS = {"redis_cls": "redisbloom.client.Client"}
BLOOM_DUPEFILTER_ERROR_RATE = 0.001
BLOOM_DUPEFILTER_CAPACITY = 100_0000

You can also set queue and bloom‑filter parameters directly in the spider class:

class MySpider(SitemapSpider):
    ...
    queue_conf: QueueConfig = QueueConfig(
        name="example", durable=True,
        arguments={"x-queue-mode": "lazy", "x-max-priority": 255}
    )
    redis_bloom_conf: RedisBloomConfig = RedisBloomConfig(
        key="example:dupefilter", error_rate=0.001, capacity=100_0000
    )

Step 3: Run the spider scrapy crawl example Check that the RabbitMQ queue and RedisBloom filter are operating correctly.

Why Scrapy-Distributed?

Existing solutions such as Scrapy-Redis and scrapy‑rabbitmq have several drawbacks: Redis set‑based deduplication consumes large memory, Redis list queues can cause backlog, RabbitMQ integrations lack full parameter control, distributed dupefilter support is missing, and they require invasive code changes.

Scrapy-Distributed solves these problems by using RedisBloom for low‑memory deduplication, providing full RabbitMQ queue configuration (including lazy‑mode), offering a flexible scheduler that can combine components, and adopting a non‑intrusive design that only needs configuration changes to make any spider distributed.

The framework is open‑source on GitHub.