How to Build a High‑Speed Sina Weibo Scrapy Spider that Crawls 13 Million Posts Daily

Spider Features

This project, similar to a QQ Space spider, mainly crawls Sina Weibo user personal information, posts, followers and followees (see the GitHub repository for details).

The code obtains Sina Weibo cookies for login; multiple accounts can be used to evade anti‑scraping measures (accounts can be purchased cheaply).

The spider targets the Sina Weibo WAP site, which has a simple structure and relatively weak anti‑scraping defenses, resulting in faster crawling though with slightly less data.

Crawling speed can reach over 13 million records per day, depending on network conditions (the author achieved this on a campus network; typical home connections may be slower).

Environment and Architecture

Development language: Python 2.7 Development environment: 64‑bit Windows 8, 4 GB RAM, i7‑3612QM CPU Database: MongoDB 3.2.0

Uses the Scrapy framework.

Download middleware randomly selects a Cookie and User‑Agent from pools and adds them to each request.

In start_requests, four requests are launched per user ID to crawl personal info, posts, followees and followers.

Newly crawled followee and follower IDs are added to the queue after deduplication.

Usage Instructions

Before starting:

MongoDB must be installed and running; no additional configuration needed.

Python must have Scrapy installed (prefer 64‑bit dependencies for a 64‑bit Python).

Additional Python modules required: pymongo, json, base64, requests.

Add your Sina Weibo login credentials to cookies.py (the file already contains two example accounts).

Scrapy settings such as download delay, log level, and concurrent requests can be adjusted in the settings file.

Running Screenshots

Database Description

SinaSpider stores crawled data in four MongoDB collections: Information , Tweets , Follows , and Fans . Below are the fields of the first two collections.

Information collection fields: _id – user ID (unique) Birthday – date of birth City – city Gender – gender Marriage – marital status NickName – Weibo nickname Num_Fans – number of fans Num_Follows – number of followees Num_Tweets – number of posts Province – province Signature – personal signature URL – personal homepage URL

Tweets collection fields: _id – composite key "userID‑tweetID" (unique) Co_oridinates – geolocation coordinates of the post (can be mapped to a location) Comment – number of comments Content – post content ID – user ID Like – number of likes PubTime – publication time Tools – device or platform used to post Transfer – number of reposts