How to Use Scrapy to Crawl Zhihu Users and Analyze Their Data

Scrapy Principle

Scrapy works like a restaurant ordering system: the engine sends requests (orders) to the internet (kitchen), receives responses (dishes), and passes the data through pipelines (delivery) to be stored.

Creating a Scrapy Project

Install Scrapy and start a new project with: $ scrapy startproject zhihuxjj Create a spider file zhihuxjj.py inside the spiders directory.

Defining the Spider

The spider sets the target user, constructs URLs for followees and user details, and yields Request objects with callback methods parse_fo and parse_user.

class ZhihuxjjSpider(Spider):
    name = 'zhihuxjj'
    allowed_domains = ["www.zhihu.com"]
    start_user = "jixin"
    followees_url = 'https://www.zhihu.com/api/v4/members/{user}/followees?...&limit=20'
    user_url = 'https://www.zhihu.com/api/v4/members/{user}?include=...'

    def start_requests(self):
        yield Request(self.followees_url.format(user=self.start_user, offset=0), callback=self.parse_fo)
        yield Request(self.user_url.format(user=self.start_user, include=self.user_include), callback=self.parse_user)

    def parse_user(self, response):
        result = json.loads(response.text)
        item = ZhihuxjjItem()
        item['user_name'] = result['name']
        item['sex'] = result['gender']
        item['user_sign'] = result['headline']
        item['user_avatar'] = result['avatar_url_template'].format(size='xl')
        item['user_url'] = 'https://www.zhihu.com/people/' + result['url_token']
        if result['locations']:
            item['user_add'] = result['locations'][0]['name']
        else:
            item['user_add'] = ''
        yield item

    def parse_fo(self, response):
        results = json.loads(response.text)
        for result in results['data']:
            yield Request(self.user_url.format(user=result['url_token'], include=self.user_include), callback=self.parse_user)
            yield Request(self.followees_url.format(user=result['url_token'], offset=0), callback=self.parse_fo)
        if not results['paging']['is_end']:
            next_url = results['paging']['next'].replace('http', 'https')
            yield Request(next_url, callback=self.parse_fo)

Item and Pipeline

Define the fields to extract in items.py and write a pipeline in pipeline.py that inserts each item into a MySQL table.

class ZhihuxjjItem(Item):
    user_name = Field()
    sex = Field()
    user_sign = Field()
    user_avatar = Field()
    user_url = Field()
    user_add = Field()

class ZhihuxjjPipeline(object):
    def process_item(self, item, spider):
        conn = dbHandle()
        cursor = conn.cursor()
        sql = "insert into xiaojiejie.zhihu(user_name,sex,user_sign,user_avatar,user_url,user_add) values(%s,%s,%s,%s,%s,%s)"
        param = (item['user_name'], item['sex'], item['user_sign'], item['user_avatar'], item['user_url'], item['user_add'])
        try:
            cursor.execute(sql, param)
            conn.commit()
        except Exception as e:
            print(e)
            conn.rollback()
        return item

Settings Adjustments

Disable ROBOTSTXT_OBEY, set a realistic DOWNLOAD_DELAY, and provide custom request headers (User-Agent and authorization token) in settings.py to avoid being blocked.

DEFAULT_REQUEST_HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
    "authorization": "oauth c3cef7c66a1843f8b3a9e6a1e3160e20"
}

Running the Spider

Execute the crawl with scrapy crawl zhihuxjj or wrap it in a main.py script and run from an IDE.

$ scrapy crawl zhihuxjj

Data Analysis Results

After several days the spider collected about 70,000 user records with a depth of 5. Simple analysis shows male users dominate (>50%), female users (~30%), and the majority of female users are located in first‑tier cities (Beijing, Shanghai, Guangzhou, Shenzhen, Hangzhou).

Conclusion

The tutorial demonstrates that with a few dozen lines of Scrapy code you can build a functional Zhihu crawler, store results in a database, and perform basic demographic analysis, while reminding readers to respect robots.txt and platform policies.