How to Build a Python Zhihu Crawler: Login, User Data, and Answer Likes

Simulated Login

To crawl Zhihu, first simulate a login using a function that sends a POST request with your credentials; the first run requires manual captcha entry, after which a cookie file and captcha image are saved for automatic future logins.

Get User Basic Info

Each Zhihu user has a unique ID; by accessing https://www.zhihu.com/people/ {userID} you can retrieve profile details such as location, industry, gender, education, vote counts, followers, and followees. The provided get_userInfo(userID) function returns a list of 19 fields including nickname, ID, location, company, position, school, major, vote counts, and follower statistics.

Get Answer Likes List

To obtain all users who liked a specific answer, first identify the answer's unique ID from its URL, then locate the voters_profile endpoint (e.g., https://www.zhihu.com/answer/5430533/voters_profile). The JSON response contains liker information and pagination URLs; each request returns up to 20 likers, including nickname, profile URL, vote count, thanks count, and question/answer counts. The script saves the data to a text file named after the answer ID, handling anonymous or deleted users by noting missing information.

Extract User Avatar

Given a user ID, a function parses the user's profile page to locate the avatar image URL, downloads the image, and saves it locally using the user ID as the filename.

Fetch All Answers of a Question

By providing a question ID, another function iterates through all answer pages, extracts the textual content of each answer (excluding images), and saves each answer to a separate text file named after the answer's author ID.

Database Storage

After gathering the data, the author stores user information in a SQLite3 database table, enabling easy retrieval and further analysis such as visualizing follow relationships among prominent users. Future work includes scaling the crawler, handling anti‑scraping measures, and exploring Scrapy or Weibo scraping.