How to Build a Zhihu Crawler with Python, ELK, and Visual Analytics

Introduction

Zhihu is a popular Chinese Q&A community built with Python, making it an attractive target for web‑scraping experiments. This guide showcases a Python‑based crawler that extracts user data from Zhihu and visualizes it using the ELK stack.

Technology Stack

Crawler: Python 2.7, requests, json, BeautifulSoup4, time Analysis tools: ELK suite (Elasticsearch, Logstash, Kibana)

IDE: PyCharm

Data Collected

The crawler retrieves partial user information from Zhihu.

Simple Visual Analysis

1. Gender Distribution

Green indicates male, red indicates female, and gray indicates unknown. The chart shows a predominance of male users.

2. Top 30 Users by Followers

The top three users are Zhang Jiawei, Li Kaifu, Huang Jixin, confirming the crawler's credibility.

3. Top 30 Users by Articles Written

Crawler Architecture

The architecture diagram is shown below.

Implementation Details

Select an active user (e.g., Li Kaifu) as the entry URL and store visited URLs in a set. Crawl each user's followees, add their URLs to another set, and filter already visited URLs. Parse personal information and save it locally.

Logstash reads the local files and forwards data to Elasticsearch; Kibana visualizes the data.

Authorization Retrieval

Open Chrome, log into Zhihu, inspect a user’s page, click “Follow”, refresh, and capture the request header’s authorization token as shown in the screenshot.

Possible Improvements

Introduce a thread pool to increase crawling speed.

Replace the in‑memory set() with Redis for URL storage.

Store crawled data in MongoDB instead of local files.

Filter users by follower count or topic participation before saving.

ELK Suite Notes

Installation details are available on the official Elastic website. A sample Logstash configuration file is shown below.

Conclusion

The harvested user data can be analyzed for geography, education, age, and more. Web crawling remains a fascinating way to extract valuable insights from the vast internet data ocean, especially in today’s content‑driven era.