Master Web Scraping with XPath: A Step‑by‑Step Scrapy Tutorial

Introduction

After learning XPath basics, this article demonstrates how to use XPath expressions within the Scrapy framework to extract various fields such as title, publication date, tags, content, likes, favorites, and comments from a sample website.

Implementation Details

1. Extract the title using any of the previously shown XPath expressions, test it in the Scrapy shell, and write the selector into the spider.

2. Extract the publication date by interacting with the page source; the element with class entry-meta-hide-on-mobile is unique and can be located directly.

3. The article’s topic tags appear below the date in the HTML structure; locate them using a suitable XPath expression.

4. After retrieving the list of tags, join them with commas using the join function and store the result in the spider.

5. The number of likes can be captured by locating the element with class vote-post-up. If multiple classes are present, use the contains() function in XPath, e.g., //span[contains(@class,"vote-post-up")].

6. Convert the extracted like count from a string to an integer with int() before further processing.

Conclusion

This tutorial builds on fundamental XPath knowledge to show practical data extraction with Scrapy, laying the groundwork for larger‑scale web crawling projects.