Master Scrapy: Extract Likes, Comments, and Content with XPath

Introduction

In the previous article we introduced how to use Scrapy XPath selectors to collect target data from web pages. This continuation focuses on extracting like counts, comment counts, and article content using XPath, regular expressions, and debugging with Scrapy shell.

Implementation Details

9. By locating the like count element we can quickly find the collection count, which has a slightly different HTML structure but the analysis method remains the same.

10. The debugging code is shown below.

11. To extract only the numeric part we apply a regular expression; the code can be debugged in PyCharm.

12. The comment count is simpler, with a dedicated tag.

13. Note that the comment tag uses an href attribute rather than a class; mismatched selectors will return empty results.

14. Like the collection count, use a regular expression to capture the numeric comment count, reusing the collection code and replacing the variable name.

15. Extracting the main article body varies by page; generally the content resides under an entry tag.

16. In Scrapy shell we can obtain the XPath expression for the article body.

17. Combining the analysis and XPath expressions yields the complete crawling code.

18. Debugging the spider shows the captured data clearly.

19. The console output confirms that the extracted values match the web page.

Conclusion

Overall, use the browser's F12 tool to inspect elements, analyze the page structure, craft XPath expressions, test them in Scrapy shell, embed them into the spider, and run or debug to obtain the final data.