How to Build a High‑Performance Novel Site Crawler with MongoDB
This article walks through building a novel‑site crawler using MongoDB, detailing how to extract category links, manage URL states across multiple processes, and handle deduplication, while sharing screenshots of the framework, database logic, and final results.
Inspired by a tutorial on crawling a top‑ranked site, the author decided to create a crawler for the novel website "Yisou" and chose MongoDB instead of MySQL for data storage.
The first step is to extract the links of each category from the ranking page and then crawl each category, as shown in the all_theme file.
The crawler constructs a list of all page URLs for each category, resulting in more than 5,000 pages.
Database operations are encapsulated to support multiprocessing and multithreading. Each URL can be in one of three states:
outstanding – waiting to be crawled
processing – currently being crawled
complete – crawl finished
When all initial URLs are outstanding , the crawler changes their state to processing as it starts, then to complete upon success; failed URLs revert to outstanding . A timer resets URLs to outstanding if a process is terminated.
The main crawler program is illustrated below.
After running, the raw results contain many duplicate entries; after deduplication, only a few hundred thousand unique books remain, which was disappointing.
Author: 蛞牛仔 (Source: http://www.jianshu.com/p/a1c5183f3f4d)
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
MaGe Linux Operations
Founded in 2009, MaGe Education is a top Chinese high‑end IT training brand. Its graduates earn 12K+ RMB salaries, and the school has trained tens of thousands of students. It offers high‑pay courses in Linux cloud operations, Python full‑stack, automation, data analysis, AI, and Go high‑concurrency architecture. Thanks to quality courses and a solid reputation, it has talent partnerships with numerous internet firms.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.