Master Scrapy Requests: Download Pages and Trigger Callbacks Efficiently

In the previous article we collected the detail page URLs of articles; the next step is to hand these URLs to Scrapy for downloading and to invoke our custom parsing function.

Specific Implementation

The Request class resides in scrapy.http and can be imported directly.

We create a Request object with the article URL as the url parameter and a callback function (e.g., parse_detail) to process the response.

The callback extracts the desired fields from the detail page; you can use CSS selectors (or XPath if preferred) inside parse_detail.

When constructing the Request, add the callback argument and reference the method with self. (e.g., callback=self.parse_detail) to avoid errors.

Often the URL in the request is relative; use response.urljoin() (or parse.urljoin()) to combine it with the base domain and obtain a full URL.

Finally, yield the Request object (e.g., yield Request(url, callback=self.parse_detail)) so Scrapy schedules it for download.

At this point we have completed the process of extracting all article URLs from the list page, handing them to Scrapy for download, and parsing each detail page. The next article will cover how to obtain and process the next‑page URL.