How to Build a Zhihu Web Crawler with Python, ELK Visualization, and Tips
This article walks through creating a Python‑based Zhihu crawler, visualizing the harvested user data with the ELK stack, and offers practical improvements such as threading, Redis URL storage, and MongoDB persistence for more efficient and scalable web scraping.
Overview
Zhihu is a real Q&A community; many Python enthusiasts use it for web‑crawling experiments. This article presents a Python 2.7 crawler built with requests, json, BeautifulSoup, and time, and shows how to analyze the collected data using the ELK suite.
Technology Stack
Crawler: python27 + requests + json + bs4 + time
Analysis tools: ELK suite
IDE: PyCharm
Data Results
The crawler fetched partial user data from Zhihu.
Simple Visual Analysis
Gender Distribution
Male users dominate Zhihu.
Top 30 Users by Followers
The top three are Zhang Jiawei, Li Kaifu, Huang Jixin, confirming the data’s credibility.
Top 30 Users by Articles Written
Crawler Architecture
The architecture uses an entry URL (e.g., an active user), stores visited URLs in a set, crawls followees, parses personal info, stores it locally, feeds data to Logstash, then Elasticsearch and Kibana for visualization.
Implementation Details
Steps to fetch a URL, parse content, and store locally are illustrated with diagrams.
Obtaining Authorization
Open Chrome, log into Zhihu, inspect the network while following a user, and capture the authorization header as shown.
Potential Improvements
Add a thread pool to increase crawling efficiency.
Replace the in‑memory set with Redis for URL storage.
Store results in MongoDB instead of local files.
Filter users by follower count (>100) or topic participation (>10) to avoid dead accounts.
ELK Suite Notes
Installation details are on the official Elastic website. An example Logstash configuration is shown below.
Conclusion
The collected user data can be analyzed for geography, education, age, and more. Web crawling is valuable for extracting insights from the vast internet data sea. This article is for learning and exchange only; all data belongs to Zhihu.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
MaGe Linux Operations
Founded in 2009, MaGe Education is a top Chinese high‑end IT training brand. Its graduates earn 12K+ RMB salaries, and the school has trained tens of thousands of students. It offers high‑pay courses in Linux cloud operations, Python full‑stack, automation, data analysis, AI, and Go high‑concurrency architecture. Thanks to quality courses and a solid reputation, it has talent partnerships with numerous internet firms.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.