Create Your First Scrapy Spider Project: Step-by-Step Structure Walkthrough

This article provides a hands-on guide to creating a first Scrapy spider project on Windows, assuming Scrapy is already installed in a virtual environment.

Prerequisites

Activate the virtual environment where Scrapy is installed. Verify the installation with pip list to confirm Scrapy appears in the package list.

Step 1: Navigate to Target Directory

Change to the desired parent directory (e.g., a demo folder) where the new project will be created.

Step 2: Create the Scrapy Project

Run the command:

scrapy startproject article

article

is the project name and can be customized. Scrapy generates the project from its built-in template located at Lib\site-packages\scrapy\templates\project within the virtual environment. The article notes that custom templates are possible but the default is sufficient for learning.

Step 3: Inspect the Generated Structure

Enter the project directory with cd article and list files using dir or tree /f for a tree view. The structure consists of three layers:

Top layer: article/ — the project root folder named after the project.

Second layer: article/ (a Python module containing all project code) and scrapy.cfg (the project-wide configuration file).

Third layer (inside the module): __init__.py — empty file that makes the directory a Python package. items.py — defines data models (Item classes) for scraped data. middlewares.py — middleware components handling request/response processing; rarely modified. pipelines.py — defines item pipelines for post-processing and storage of scraped data. settings.py — central configuration: pipeline activation, download delays, database settings, etc. spiders/ — directory holding spider implementations (crawling logic) plus its own __init__.py.

Step 4: Alternative Views

The same file tree is visible in Windows Explorer and can be imported into PyCharm for a clearer IDE-based view.

Step 5: Examine Key Configuration

The article shows the default content of settings.py as an example; other files are not detailed here but follow the same pattern.

The tutorial concludes by previewing upcoming advanced Scrapy topics.