How to Build a High‑Performance Novel Site Crawler with MongoDB and Multithreading

Inspired by a tutorial on crawling a top novel site, the author decided to crawl the "Yisou" novel website using MongoDB instead of MySQL.

First, the overall code framework is shown.

The crawler starts by extracting every category link from the ranking page, then follows each link to crawl the content. The all_theme file is examined first.

The running result shows the book category.

From this, the crawler constructs all page‑link URLs for each category, creating the entry points for the spider – over 5,000 pages in total.

Database operations are encapsulated to support multiprocessing and multithreading. Each process must know which URLs have been crawled and which remain. Two status flags are defined for every URL:

outstanding – waiting to be crawled

processing – currently being crawled

complete – crawl finished

The state transitions are simple: all initial URLs are marked outstanding ; when crawling starts, the status changes to processing ; upon success it becomes complete ; failed URLs revert to outstanding . A timer resets stalled URLs back to outstanding after a timeout.

The main crawler program is then presented.

Running the crawler yields many duplicate entries; after deduplication, only a few hundred thousand unique books remain, which the author finds disappointing.