Build a Fast Python Web Scraper for Novel Rankings – Step by Step

Goal

Retrieve the name and link of every novel listed on the ranking page http://www.qu.la/paihangbang/ .

Observing the Webpage Structure

The site groups each category inside a consistent HTML container, making it easy for a crawler to locate all novel links and store them in a list.

List Deduplication Trick

Novels may appear in multiple categories, causing duplicate entries. Using a set (or converting a list to a set) removes duplicates efficiently.

Code Implementation

Modular, functional programming is recommended: each independent task should be a separate function.

1. Request Headers

2. Get Ranking Novels and Links

Iterate each category, write "novel name + URL" to a file, and collect URLs into a list.

3. Get All Chapter Links of a Single Novel

Fetch each chapter URL and create a local file for the novel.

4. Retrieve Page Content and Save Locally

A common issue is that scraped HTML contains formatting tags like <br/>. Replace them with line breaks:

html = get_html(url).replace('<br/>', '
')

5. Main Function

6. Output Results

Performance and Next Steps

Estimated timings: one article ≈ 0.5 s, a novel (~1000 pages) ≈ 8.5 minutes, the full 60‑item ranking list ≈ 8.5 hours. To speed up, consider multithreading or, better yet, refactor the code using the Scrapy framework, which can boost crawling speed by dozens or hundreds of times.