How to Scrape WeChat Moments with Python Scrapy: Step‑by‑Step Guide

1. Obtain WeChat Moments Data Entry

Follow the public account "出书啦", click "创作书籍" → "微信书", add the provided QR‑code friend, and wait for the generated WeChat book link. Ensure your Moments are set to "全部开放" (public).

2. Create Scrapy Project

Install Scrapy, then run:

scrapy startproject weixin_moment

Enter the project directory and generate a spider:

scrapy genspider 'moment' 'chushu.la'

The resulting folder structure is shown in the accompanying screenshot.

3. Analyze Webpage Data

Open the WeChat book page in Chrome, press F12, go to the Network tab, and enable "Preserve log". The initial request is a GET returning JSON data. Navigating by month triggers POST requests with changing payload parameters, and the response JSON contains the Moments under the /data node.

4. Code Implementation

Modify items.py to define fields for the post content and its publish date.

Update the spider file moment.py :

Import the item class.

Implement start_requests to request the exported WeChat book URL.

Parse the navigation JSON in parse, handling byte‑to‑string conversion.

Construct POST parameters as strings (year, month, index) and include required headers such as Referer.

Define parse_moment to extract each Moment from the JSON and yield the populated item.

Enable the item pipeline in settings.py by uncommenting the relevant line.

Run the spider:

scrapy crawl moment -o moment.json

If the output file shows garbled characters, re‑run with UTF‑8 encoding:

scrapy crawl moment -o moment.json -s FEED_EXPORT_ENCODING=utf-8