Create Your First Scrapy Spider: genspider Command, PyCharm Import & Interpreter Setup

After creating a Scrapy project named article, the framework prompts you to create a spider using a built-in template. First, run cd article to enter the project directory, then execute scrapy genspider jobbole blog.jobbole.com. This command uses Scrapy's basic template to generate a spider named jobbole targeting the domain blog.jobbole.com. The output confirms the spider was created at article.spiders.jobbole.

Run tree /f to verify the file structure. Besides the initial project files, a new jobbole.py file appears under the spiders folder. While custom templates are possible, the built-in basic template is sufficient for most use cases.

Next, import the project into PyCharm: choose File → Open and select the project folder. If jobbole.py does not appear under the spiders folder in PyCharm, right-click the spiders folder and select Synchronize spider to refresh the view.

Open jobbole.py to inspect the generated code. The template pre-fills three key attributes: name (the spider's unique identifier), allowed_domains (restricts crawling to blog.jobbole.com), and start_urls (the initial URLs to crawl).

Finally, verify the Python interpreter. In PyCharm, open Settings and search for interpreter . If the displayed interpreter is not the project's virtual environment, click the gear icon next to Project Interpreter , choose Add Local , and select the virtual environment's Python executable. This ensures the project runs with the correct dependencies.

The tutorial uses the Jobbole article site as a running example. The GitHub repository https://github.com/cassieeric is referenced for additional crawler projects.