How I Built a High‑Performance Novel Site Crawler with MongoDB
Inspired by a tutorial, I created a MongoDB‑backed crawler for the Yisou novel website, extracting category links, managing URL states across multiple processes, handling millions of pages, and finally deduplicating the results to obtain a clean collection of books.
Inspired by a tutorial on crawling the "Peak" site, I decided to build my own crawler for the Yisou novel website using MongoDB instead of MySQL.
First, I extracted the links for each category from the ranking page, then traversed each category's pages (over 5,000 pages) as shown in the architecture diagram.
To manage URLs across multiple processes and threads, I defined three states for each URL: outstanding (waiting to be crawled), processing (currently being crawled), and complete (crawling finished). Failed URLs are reset to outstanding , and a timeout mechanism resets stalled URLs.
outstanding: waiting to be crawled
processing: currently being crawled
complete: crawling finished
The main crawler program coordinates these states, launches workers, and stores the fetched data into MongoDB.
After running the crawler, the raw result contained many duplicate entries; after deduplication only a few hundred thousand unique books remained.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
MaGe Linux Operations
Founded in 2009, MaGe Education is a top Chinese high‑end IT training brand. Its graduates earn 12K+ RMB salaries, and the school has trained tens of thousands of students. It offers high‑pay courses in Linux cloud operations, Python full‑stack, automation, data analysis, AI, and Go high‑concurrency architecture. Thanks to quality courses and a solid reputation, it has talent partnerships with numerous internet firms.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.