How to Build Your First Scrapy Project on Windows: Step‑by‑Step Guide

First, activate a Python virtual environment on Windows; if you are unfamiliar with creating or activating one, refer to the linked tutorials. Inside the environment run pip list to verify that Scrapy is installed successfully (see screenshot).

Next, move the Scrapy project folder into a demo directory and navigate up one level to keep the workspace organized.

Step 1: Create the Scrapy project

Run the command scrapy startproject article. article is the project name and can be changed as needed. Scrapy generates a template based on its internal files located under the Python site‑packages directory.

Step 2: Enter the project directory

Change into the newly created folder with cd article. Use dir or tree /f to view the generated file tree, which clearly shows the project’s structure.

Project structure overview

The top‑level article folder is the project name. Inside it there is a subfolder also named article (the Python module) and a scrapy.cfg configuration file.

Within the module folder you will find five files and one subfolder: __init__.py: empty file that marks the directory as a Python package. items.py: defines the data items to be scraped. middlewares.py: contains middleware components (usually left unchanged). pipelines.py: processes and stores scraped data. settings.py: project settings such as pipelines, crawl rate, and default request headers. spiders folder: holds spider scripts that implement the crawling logic, plus its own __init__.py.

Step 3: View the project in Windows Explorer

The Windows file explorer shows the same structure, confirming that the project was created correctly.

Step 4: Open the project with PyCharm

Import the article project into PyCharm for a clearer view of the files and their contents.

Step 5: Inspect settings.py

The screenshot displays the default settings.py content; other files follow similar default templates.

With these steps completed, you have a functional Scrapy project ready for further development and customization.