How to Build a Python Zhihu Crawler: Login, User Data, and Answer Likes Extraction

Simulated Login

To scrape Zhihu you must first simulate a login because many pages are inaccessible without authentication. The author provides a login function that uses the user's credentials and handles the captcha on the first run, storing cookies and the captcha image for subsequent automated logins.

Fetching User Basic Information

Each Zhihu user has a unique ID (e.g., marcovaldong). By accessing https://www.zhihu.com/people/{userID} you can retrieve profile details such as location, industry, gender, education, vote counts, follow statistics, etc. The function get_userInfo(userID) returns a list of 19 fields including nickname, ID, residence, industry, gender, company, position, school, major, upvotes, thanks, questions asked, answers given, articles, collections, edits, followers, followees, and profile views.

Extracting All Likers of an Answer

The article explains how to obtain the list of users who liked a specific answer. Each question and answer has a unique numeric ID extracted from its URL. By inspecting the network request for the "5321 people liked" button, you can find a URL like https://www.zhihu.com/answer/5430533/voters_profile, where 5430533 is the ID used to fetch the likers.

Using requests to GET this endpoint returns a JSON payload containing liker information. The payload is paginated, with each request returning up to 20 users and a link to the next page. The script extracts each liker’s nickname, profile URL (user ID), upvote count, thank count, question count, and answer count. Anonymous or deleted users are noted as missing.

Downloading User Avatars

Given a user’s unique ID, a function parses the user’s profile page to locate the avatar image URL, downloads the image, and saves it locally using the user ID as the filename.

Fetching All Answers of a Question

By providing a question’s unique ID, another function retrieves all answers under that question. Only the textual content is saved (images are omitted), and each answer is written to a separate .txt file named after the answerer’s ID.

Storing Data in a Database

After gathering the data, the author demonstrates storing user information in a SQLite database. The basic schema creates a table to hold the 19 fields returned by get_userInfo. With SQLite familiar, the next steps include bulk collection of user and follower data and visualizing the follow network, possibly using Scrapy for larger-scale crawling.

Anti‑Scraping Considerations

The author notes that Zhihu may block frequent requests, prompting captcha challenges. Future improvements will involve rate‑limiting, rotating proxies, and other anti‑anti‑scraping techniques.