How to Run Multiple Scrapy Spiders Efficiently: Cmdline, CrawlerProcess, and CrawlerRunner

1. Run a spider from the command line

First, create the spider file baidu.py (see image). The spider can be launched in two ways: directly with scrapy crawl baidu or by executing the script with python baidu.py (see screenshot).

2. Run a spider inside a Python file

Three approaches are shown:

Cmdline method : use cmdline.execute to start the spider (image).

CrawlerProcess method : instantiate CrawlerProcess and call process.start() (image).

CrawlerRunner method : create a CrawlerRunner and schedule the spider (image).

3. Run multiple spiders in the same project

Add a second spider, e.g., sina.py, to the spiders directory (image).

Running both spiders with the cmdline method fails because the process exits after the first spider finishes (image).

Using CrawlerProcess allows both spiders to run concurrently; the middleware is initialized only once, and requests are sent in parallel (image).

Running spiders sequentially with CrawlerRunner also loads the middleware a single time but executes spiders one after another, reducing interference; this approach is recommended by the official documentation (image).

Summary

The cmdline.execute method offers the simplest configuration for running a single spider file repeatedly, while CrawlerProcess and CrawlerRunner provide more robust solutions for executing multiple spiders with minimal middleware duplication.