Step‑by‑Step Scrapy Guide: Crawl All Pages of a Blog Automatically

This article continues a series on Python web crawling with Scrapy, focusing on extracting all article URLs from a list page and iteratively crawling subsequent pages to collect complete site content.

Step 1: Set start_urls to the list‑page URL

The URL of the article list (not a single article) is placed in start_urls.

Step 2: Modify parse() to handle two tasks

1) Extract every article URL on the current page and schedule a request to parse its content. 2) Extract the next page URL, schedule it, and let parse() process it again.

Step 3: Analyze the page structure

Using browser developer tools you can see that each list page contains 20 article links under the element with id="archive". The links are peeled layer by layer like an onion.

Step 4: Locate the article detail link

Clicking the dropdown arrow reveals that the detail‑page URL is not deeply hidden; it can be captured directly.

Step 5: Test selectors in Scrapy shell

Enter the following command in the shell to open the debugging window and try CSS selectors: scrapy shell "<list_page_url>" Using a::attr(href) extracts the href attribute of each link – a handy trick for gathering URLs.

Step 6: Verify extraction

After running the selector, the first page’s 20 article URLs are obtained. These URLs are then handed to Scrapy for downloading, and the custom parse function processes each article.

Further steps (scheduling the next page, handling pagination) will be covered in the next article.

Conclusion : This guide lays the theoretical foundation for using Scrapy to crawl a multi‑page site, preparing you for full‑site data extraction in subsequent tutorials.