Mastering WeChat Moments Scraping with Scrapy: Step-by-Step Code Guide

In the previous article we introduced the theory of using Python Scrapy to crawl WeChat Moments; this article provides the practical implementation.

1. Modify items.py

Define two fields, date and content, to store the moment date and text.

2. Update the spider ( moment.py )

Import the WeixinMomentItem class, adjust start_requests to send the correct POST parameters, and ensure the response is decoded from bytes to string.

3. Parse the response

Implement the parse method to extract navigation data; note the need to convert bytes to str, use string parameters for year/month/index, and include the Referer header to avoid anti‑hotlink errors.

The response from the page is bytes and must be converted to str before parsing.

POST parameters (year, month, index) must be strings; otherwise the server returns 400.

The request header must contain a Referer to pass anti‑hotlink checks.

Other implementations of the request construction are also possible.

4. Extract moment data

Define parse_moment to load the JSON response and retrieve the desired fields.

5. Enable the pipeline

Uncomment ITEM_PIPELINES in settings.py so that items are processed.

6. Run the spider

scrapy crawl moment -o moment.json

The command generates a moment.json file containing the scraped data.

7. Fix encoding issues

If the JSON appears garbled, delete the file and rerun with UTF‑8 encoding:

scrapy crawl moment -o moment.json -s FEED_EXPORT_ENCODING=utf-8